Designing Reddit’s Unified Media Metadata Store: 100k RPS at 17ms p99 Latency

Reddit processes billions of posts containing rich media—images, videos, GIFs, and embedded media. Each media asset requires critical metadata such as playback URLs, transcoding profiles, bitrates, thumbnail URLs, and dimensional properties.

Historically, this metadata was scattered across multiple fragmented databases, leading to operational complexity, inconsistent query interfaces, and difficult cross-media analytics. Reddit engineered a consolidated, high-throughput metadata store powered by AWS Aurora PostgreSQL capable of handling 100,000 requests per second (RPS) with a p99 latency of 17ms—notably achieved without a read-through cache.

---

1. System Architecture Overview

Reddit selected AWS Aurora PostgreSQL as the single source of truth for all media metadata. The end-to-end architecture consists of an API service layer, connection management, the Aurora relational engine, and auxiliary asynchronous streams.

flowchart TD
    Client[Clients / Services] -->|HTTP / gRPC| API[API Service Layer]
    API -->|High-throughput Connection Pool| PB[PgBouncer]
    PB -->|Backend Process Connections| Aurora[(AWS Aurora PostgreSQL)]
    
    subgraph Data Tier
      Aurora --> JSONB[(JSONB Metadata Table)]
      Aurora --> Partitions[Time-based Partitions via pg_partman]
    end

Why PgBouncer is Mandatory for PostgreSQL

PostgreSQL allocates an entire operating system process (via fork()) per client connection rather than a lightweight thread. Each process consumes dedicated memory (work memory, connection state buffers) and incurs operating system context-switching overhead.

At 100,000 RPS across hundreds of application worker pods, establishing direct database connections would quickly saturate PostgreSQL’s process limits and exhaust server memory. Reddit placed PgBouncer in front of Aurora to implement connection pooling, multiplexing thousands of incoming application connections into a tightly bounded pool of persistent server backend processes.

---

2. Zero-Downtime Data Migration Architecture

Migrating several terabytes of live metadata from multiple legacy databases to a unified store while serving 100,000 live RPS requires a phased, zero-downtime migration pipeline.

sequenceDiagram
    autonumber
    participant App as API Clients
    participant Src as Source DBs
    participant Kafka as Kafka (CDC)
    participant Worker as Sync / Validation Consumer
    participant Unified as Aurora Metadata DB

    App->>Src: Write New Metadata
    Src-->>Kafka: Stream Mutations (CDC)
    Kafka->>Worker: Consume Event
    Worker->>Worker: Validate Idempotency & Conflict Checks
    Worker->>Unified: Upsert Validated Record
    Worker-->>Unified: Log Discrepancies (Audit Table)

Migration Phases

1. Dual Writes: Application write paths are updated to write to both the legacy source databases and the new Aurora metadata pipeline.
2. Historical Backfill: A background batch migration reads older historical records from source databases and populates the Aurora metadata store.
3. Dual Reads (Shadow Validation): Read queries are executed against both the legacy databases and Aurora. Results are diffed asynchronously to verify data correctness and parity.
4. Read Cutover & Ramp-Up: Production read traffic is progressively shifted (e.g., 10% → 50% → 100%) to the unified Aurora store once parity validation passes.

---

3. Mitigating Migration Race Conditions

Dual-write patterns in distributed systems introduce two major failure modes:

Problem 1: Partial Write Failures

A write to the source database may succeed while the write to the unified database fails, or vice versa, causing immediate state drift.

Problem 2: Stale Backfill Overwriting Fresh Writes

If an active user updates an existing media record during the migration window, a race condition occurs:

1. User writes updated metadata ($v_2$) to the new store.
2. An asynchronous backfill job reads the old metadata ($v_1$) from the source database.
3. The backfill writes $v_1$ to the new store, overwriting $v_2$ with stale data.

The Kafka Change Data Capture (CDC) Solution

To eliminate dual-write hazards, Reddit leveraged Kafka-based Change Data Capture (CDC):

• Mutations from source stores are captured and published into Kafka topics.
• Dedicated consumers process change events sequentially based on partition keys (e.g., media_id).
• Validation Logic: Before applying updates to the unified store, consumers verify timestamps, revision vectors, and record existence.
• Discrepancy Reporting: When conflicting or out-of-order writes are detected, the consumer routes the record to an inconsistency audit table inside Aurora for engineers to inspect and reconcile, preventing silent data corruption.

---

4. Storage Pattern: PostgreSQL as a NoSQL Document Store

Rather than enforcing strict Third Normal Form (3NF) relational normalization across diverse media entities (videos, images, audio, embeds), Reddit modeled the metadata using PostgreSQL’s native JSONB data type.

CREATE TABLE media_metadata (
    media_id VARCHAR(64) PRIMARY KEY,
    account_id VARCHAR(64) NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL,
    metadata JSONB NOT NULL
);

Advantages of the JSONB Pattern

• Schema Flexibility: Videos have codecs, bitrates, and durations; images have resolutions and color profiles. JSONB accommodates dynamic schemas without structural DDL migrations.
• Binary JSON Performance: Unlike standard JSON (which stores raw text and requires re-parsing on read), JSONB decomposes data into parsed binary representations. It supports indexed access, nested lookups, and fast serializations.
• Key-Value Simplicity: Reddit effectively turned Aurora PostgreSQL into an enterprise-grade, ACID-compliant document store, optimizing for single-key lookups.

---

5. Scaling Large Data Volumes: Declarative Partitioning with pg_partman

As billions of media items accumulated, a single monotonic table would have caused index bloating and degraded B-tree traversal times. Reddit used PostgreSQL table partitioning managed by the pg_partman extension.

How pg_partman Operates

• Declarative Partitioning: Supports automatic partition management based on time or numeric sequential ranges.
• Lifecycle Automation: Instead of manually executing CREATE TABLE ... PARTITION OF statements, engineers define a template and a retention policy.
• Cron-driven Maintenance: A scheduled weekly cron job invokes pg_partman maintenance functions:
  • Pre-creates upcoming partition tables ahead of time.
  • Enforces retention schedules by archiving or dropping partitions that exceed defined time windows.

-- Example pg_partman registration pattern
SELECT partman.create_parent(
    p_parent_table => 'public.media_metadata',
    p_control => 'created_at',
    p_type => 'native',
    p_interval => 'daily',
    p_premake => 7
);

Partition pruning allows queries with time ranges or partition keys to route directly to relevant sub-tables, keeping working index sets fitting entirely inside PostgreSQL’s shared memory (shared_buffers).

---

6. Performance Benchmarks (Without a Read Cache)

Despite omitting a distributed read-through caching tier (such as Redis or Memcached), Reddit’s unified metadata store produced exceptional latency figures under sustained live traffic:

Metric	Latency
Throughput	100,000 requests/sec
p50 Latency	2.5 ms
p90 Latency	4.7 ms
p99 Latency	17.0 ms

Why No Read Cache?

1. Cache Invalidation Complexity: Eliminating a cache eliminated cache-aside stampedes, invalidation drift, and stale read states during data updates.
2. Aurora Memory Utilization: By pairing JSONB single-row lookups with connection pooling via PgBouncer and partition pruning via pg_partman, Aurora efficiently served requests directly out of buffer cache memory, achieving single-digit millisecond latency natively.

---

7. Summary & Architectural Lessons

• Process vs. Thread Model: PostgreSQL’s process-per-connection architecture makes connection pooling tools like PgBouncer mandatory for high-concurrency architectures.
• Relational Engines as Document Stores: By utilizing JSONB, relational databases can handle polymorphic, rapidly evolving payloads without sacrificing ACID safety or indexing capabilities.
• CDC Prevents Dual-Write Drift: Never rely on dual application writes for zero-downtime database migrations. Decouple writes via Kafka change streams and consumer validation to resolve race conditions and out-of-order writes.
• Partition Early: Extensions like pg_partman automate partition management and keep B-tree indexes compact enough to reside in memory, delivering sub-20ms p99 latencies under extreme load without an auxiliary cache layer.