Scaling Booking.com’s Highly Available User Review System

On travel platforms like Booking.com, user reviews are not merely a supplemental feature—they are the top of the conversion funnel. Because reviews on Booking.com require verified stays, they are perceived as highly authentic. Travelers rely heavily on this authentic feedback to make booking decisions. If the review service suffers an outage or severe latency spikes, booking conversion drops immediately.

Designing a service that serves hundreds of millions of reviews under extreme concurrency with strict latency constraints requires deliberate trade-offs across storage, caching, data partitioning, and failover topologies.

---

1. Requirements and Scale Estimation

Operational Metrics

• Peak Read Concurrency: 10,000+ requests per second (RPS).
• Latency Budget: P99 response time strictly under 50 milliseconds.
• Availability: Multi-datacenter high availability (near-zero downtime even during regional/AZ outages).

Storage Estimation

• Total Reviews: ~250 million reviews.
• Payload Structure: Objective scores (1–5 scale), parameter-based scores (cleanliness, location, staff), and free-form textual feedback.
• Average Size per Review: ~2 KB.
• Total Active Working Set:

$\text{Total Raw Data} = 250\text{ million} \times 2\text{ KB} = 500\text{ GB}$

While 500 GB is small enough to fit within the memory boundaries of a few high-end machines, the concurrency (10,000 QPS) and latency (P99 < 50ms) requirements dictate that the architecture cannot rely on a single database instance or naive query patterns.

---

2. High-Level Architecture Overview

The review infrastructure consists of stateless application services, distributed caches, and a sharded relational database layer backed by cross-zone replicas.

flowchart TD
    Client[Client Traffic] --> RS[Review Service Instances]
    RS --> Cache[(Distributed Cache: Redis/Memcached)]
    RS --> Router[Consistent Hashing Routing Layer]
    
    subgraph Shard_1 [Shard 1]
        M1[(Master DB - AZ 1)] --> R1_AZ1[(Read Replica - AZ 1)]
        M1 -.->|Async Replication| R1_AZ2[(Read Replica - AZ 2)]
    end

    subgraph Shard_2 [Shard 2]
        M2[(Master DB - AZ 1)] --> R2_AZ1[(Read Replica - AZ 1)]
        M2 -.->|Async Replication| R2_AZ2[(Read Replica - AZ 2)]
    end

    Router --> Shard_1
    Router --> Shard_2

Core System Components

1. Review Service: A stateless REST-based microservice responsible for orchestrating queries, computing consistent hashing routing, and serving read/write endpoints.
2. Centralized Caching Layer: High-throughput in-memory stores (e.g., Redis, Memcached, or Couchbase) that store hot entity reviews and summary aggregates.
3. Relational Database Cluster (MySQL): Sharded across multiple primary nodes utilizing pre-materialized views to bypass expensive join and aggregation operations at read time.
4. Multi-AZ Replicas: Read replicas provisioned both in the primary Availability Zone (for local read scale) and secondary Availability Zones (for disaster recovery and regional resilience).

---

3. Storage Optimization: Pre-Materialized Views & Caching

To hit a P99 latency target below 50ms at 10,000 RPS, querying and recalculating review scores dynamically on raw transactional tables is non-viable.

Pre-Materialized Views

A user viewing an accommodation typically requires:

• Average scores aggregated over categories (cleanliness, location, comfort).
• Filtered reviews (e.g., family vs. solo travelers, language preference).
• Recent reviews ordered chronologically.

Instead of computing these aggregation queries on the fly using SQL JOIN and GROUP BY operations, Booking.com uses materialized views. The raw transactional writes update background tables or triggers that maintain pre-aggregated summary records. When older reviews (which fall out of the cache) are queried, the service hits these pre-computed views with single-row lookups.

Two-Tier Read Path

1. Tier 1 (Cache): The top 10–20 most recent and most relevant reviews, along with the aggregate metadata for an accommodation, are loaded directly from cache.
2. Tier 2 (Materialized DB Lookups): When a user paginates deeper into the review list, queries fall back to read replicas that leverage the pre-materialized views.

---

4. Sharding and the Routing Challenge

Because a single relational database cannot handle 10,000 QPS of mixed workloads, the data is partitioned across multiple database shards. The natural sharding key is accommodation_id (or hotel_id), ensuring that all reviews for a single hotel reside on the same shard.

The Problem with Modulo Hashing

A standard partition strategy routes traffic using a simple modulo hash:

$\text{Target Shard} = \text{Hash}(\text{accommodation\_id}) \pmod N$

Where $N$ is the number of active database shards.

• The Operational Bottleneck: When traffic spikes and a new shard is added ($N \to N+1$), or when a node is removed, the modulo denominator changes. Consequently, nearly all keys ($N / (N+1)$) remap to different shards.
• The Cost: Resharding 500 GB across a running production cluster under peak load requires mass repartitioning and massive network I/O, leading to potential cache stampedes, downtime, or performance degradation.

The Solution: Consistent Hashing

Booking.com embeds a Consistent Hashing routing layer directly into the Review Service. Consistent hashing maps both nodes and keys to a fixed logical ring space (e.g., $0$ to $2^{32}-1$).

flowchart LR
    subgraph Ring Space [Logical Hash Ring: 0 to 2^32 - 1]
        A[Node 1] -->|Token Range| B[Node 2]
        B -->|Token Range| C[Node 3]
        C -->|Token Range| A
    end
    Key[Review Key] -.->|Binary Search| B

What Consistent Hashing Actually Solves

Consistent hashing is not a silver bullet that magically distributes data—it simply resolves data ownership: given an accommodation_id, it deterministically identifies which node is responsible for that key.

When a new node is added or removed, consistent hashing guarantees minimal data movement. Only $K/N$ keys need to be reassigned (where $K$ is the total number of keys and $N$ is the number of nodes), rather than reshuffling the entire database.

---

5. Practical Resizing Workflow in Production

Consistent hashing dictates where data should live, but the distributed systems engineering team must orchestrate the physical migration safely without dropping traffic.

Node Addition Sequence

1. Allocate the Node in the Ring: Calculate the token positions for the new node on the consistent hashing ring (often using virtual nodes for balanced distribution).
2. Identify Impacted Ranges: Determine the exact subset of keys that will be transferred from the adjacent peripheral nodes to the new node.
3. Background Replication: Backfill the historic data from the existing nodes to the new node in the background.
4. Catch Up Replication Logs: Stream mutations (CDC / replication binlogs) until the new node is fully caught up with the master.
5. Ring Configuration Propagation: Atomically notify the Review Service instances of the updated hash ring (e.g., via ZooKeeper, Consul, or internal configuration management).
6. Shift Traffic: The Review Service swaps its internal binary search array to point to the new hash ring, seamlessly routing subsequent traffic to the newly integrated shard.

---

6. High Availability and Cross-AZ Failover

To ensure resiliency against hardware faults and datacenter failures, each shard is deployed with a multi-availability-zone (Multi-AZ) topology.

Asynchronous Replication Strategy

• Local AZ Replica: The primary node replicates changes to a co-located read replica within the same AZ to handle local read traffic with minimal replication lag.
• Cross-AZ Replica: The primary also replicates data asynchronously to a replica located in a secondary AZ (or separate geographical region).

[ Primary DB (AZ-1) ]
       │
       ├──────── (Async Replication) ───────► [ Read Replica (AZ-1) ]
       │
       └──────── (Async Cross-AZ Replication) ─► [ Read Replica (AZ-2) ]

Why Asynchronous Replication?

Synchronous replication across availability zones forces the write path to wait for cross-datacenter round-trip times (network RTT). This introduces unpredictable latency spikes and degrades write throughput. Using asynchronous replication maintains low write latencies while accepting a marginal, bounded replication lag for reads.

Disaster Recovery

If an entire availability zone goes offline:

1. Read traffic automatically shifts to the read replica in the secondary AZ.
2. The secondary replica can be promoted to master if the primary datacenter suffers an unrecoverable failure, preserving business continuity.

---

Key Takeaways

Design Dimension	Booking.com Approach	Primary Trade-off / Benefit
Data Partitioning	Sharding by accommodation_id via Consistent Hashing	Localizes all reviews for a property; limits data movement during cluster resizing to $O(1/N)$.
Read Latency Optimization	Two-tier cache + Pre-materialized MySQL views	Hits sub-50ms P99 latency by removing runtime multi-table joins.
High Availability	Master-Replica with Cross-AZ failover	Isolates availability failures from regional datacenter outages.
Replication Mode	Asynchronous replication	Prioritizes write-path latency over strict cross-zone synchronous consistency.