How PayPal's JunoDB Scales Horizontally: Architecture, Partitioning, and Routing

JunoDB is PayPal’s open-source, highly scalable, distributed key-value store. It serves as the foundational backbone for some of PayPal’s most mission-critical workloads, including risk analysis, fraud prevention, user authentication, and core financial transaction processing. At peak production, the system processes upwards of 100 billion requests per day across hundreds of nodes.

To operate at this scale, a database cannot simply be “scaled up” with larger hardware; it must scale out horizontally across compute, networking, and storage. This article explores how JunoDB handles horizontal scaling, focusing on its connection-handling proxy tier and its two-tier data partitioning model.

---

The Three Dimensions of Scaling JunoDB

Designing for horizontal scalability requires isolating what needs to scale. In JunoDB, scale is driven by three distinct pressures:

1. Inbound Connection Pressure (Networking): As PayPal’s ecosystem grew into thousands of microservices, each service required concurrent connections to the database. Because JunoDB uses persistent connections between clients, proxies, and storage nodes for minimal latency, individual machine TCP connection limits (file descriptor caps, socket buffers) quickly become a bottleneck.
2. Storage Capacity (Data Volume): Massive transaction histories, risk sessions, and authentication tokens require storage that extends beyond a single node’s disk and memory capacity.
3. Query Throughput (Compute/IOPS): Reads and writes must be distributed across hundreds of storage nodes so no individual node becomes a CPU or I/O bottleneck.

+-------------+       +---------------+       +------------------+       +-----------------+
| Juno Client | ----> | Load Balancer | ----> |   Juno Proxy     | ----> | Storage Server  |
|    (SDK)    |       |   (Ingress)   |       | (Stateless Pool) |       | (Stateful Shard)|
+-------------+       +---------------+       +------------------+       +-----------------+

---

1. Scaling Network Connections: The Stateless Proxy Tier

To decouple client connection scaling from storage servers, JunoDB introduces a dedicated Juno Proxy layer behind standard network load balancers.

The Challenge of Connection Bloat

If microservices connected directly to storage servers, adding a new microservice would multiply the persistent connections across every database node. Storage servers would spend critical CPU cycles handling TLS handshakes, connection multiplexing, and idle socket maintenance rather than reading and writing data.

The Solution: Stateless and Equal Proxies

• Client to Proxy: Clients use the Juno SDK to open persistent TCP connections to an ingress load balancer, which distributes them across a pool of Juno Proxies.
• Stateless Design: Every Juno Proxy instance is identical and stateless. It does not own data and holds no persistent partition states locally. Instead, each proxy maintains persistent backend connections to all storage servers in the cluster.
• Horizontal Scaling: When client connections spike, operators simply spin up more Juno Proxy instances behind the load balancer. Because proxies are interchangeable, persistent connections can re-balance across new proxy instances without data re-partitioning or cluster downtime.

---

2. Scaling the Storage Tier: Fixed Shards vs. Dynamic Servers

Scaling stateful storage is fundamentally harder than scaling stateless proxies. Adding or removing storage nodes requires moving data while serving real-time traffic.

Many distributed architectures try to apply consistent hashing to every incoming key. JunoDB avoids this complexity by separating data-to-shard assignment from shard-to-server placement.

Step 1: Fixed Shard Count (Logical Partitioning)

When a JunoDB cluster is initialized, the total number of partitions—termed shards—is set to a fixed number (e.g., $1024$ shards). This shard count never changes throughout the lifetime of the cluster.

Because the number of shards is static, mapping a key to a shard does not require consistent hashing. Instead, JunoDB applies a standard deterministic modular hash:

$\text{Shard ID} = \text{MurmurHash3}(\text{Key}) \pmod{\text{Total Shards}}$

Why Not Consistent Hashing for Keys?

Consistent hashing is valuable when the target bucket count changes dynamically. Because the logical shard count ($1024$) is immutable, a direct modular arithmetic operation provides uniform key distribution, instant $O(1)$ computation, and zero topology lookup overhead.

Key: "user_12345" 
       │
       ▼
[ MurmurHash3 ]
       │
       ▼
Hash: 0x8F34A19B
       │
       ▼
  mod 1024  ────────> Assigned to Shard 412 (Fixed)

---

Step 2: Consistent Hashing for Shard-to-Node Mapping (Physical Placement)

While the number of shards is fixed, the number of physical Storage Servers is elastic. Nodes may fail, get decommissioned, or be added to expand capacity.

To map fixed shards to dynamic storage nodes, JunoDB uses Consistent Hashing:

1. Storage servers are placed at various token positions along a consistent hash ring.
2. Each of the $1024$ fixed shards is hashed onto the ring.
3. A shard is owned by the first storage server encountered moving clockwise from the shard’s position on the ring.

                     Storage Server A
                        [Token: 100]
                        /          \
                       /            \
        Shard 102     /              \    Shard 412
                     |                |   (Owned by B)
                     |    HASH RING   |
                     |                |
        Shard 850     \              /
                       \            /
                        \          /
                     Storage Server B
                        [Token: 600]

When a new storage node is added:

• It occupies positions on the hash ring.
• It takes over only a fraction of shards from its immediate neighbors.
• The vast majority of shard-to-server assignments remain untouched, guaranteeing minimal data movement during rebalancing.

---

3. Request Routing and Metadata Propagation

To execute a read or write operation, the Juno Proxy routes requests through the following pipeline:

Client Request: GET /key/session_9981
       │
       ▼
1. Compute MurmurHash3("session_9981") % 1024
   └──> Determines Target: Shard 88
       │
       ▼
2. Query Cluster Topology Map (Cached from etcd)
   └──> Consistent Hashing Ring indicates Shard 88 is on Storage Server 14
       │
       ▼
3. Forward request over existing persistent connection to Storage Server 14

Topology Management via etcd

• The mapping of Shard $\rightarrow$ Storage Server is stored as authoritative configuration state in etcd.
• When storage nodes are added, removed, or failover occurs, the updated ring mapping is written to etcd.
• Juno Proxies watch etcd and receive strongly consistent notifications of topology changes, updating their internal routing tables instantly without dropping client connections.

---

4. Data Movement via Micro-Shards

Moving an entire shard consisting of gigabytes of data during rebalancing could saturate network interfaces and cause latency spikes.

To maintain predictable latency during node addition:

• Each logical shard is internally subdivided into smaller units called micro-shards.
• When a shard is migrated to a new storage server, the source server transfers data incrementally, micro-shard by micro-shard.
• This granular transfer prevents head-of-line blocking on storage IO, preserves predictable p99 latencies, and allows rebalancing to proceed safely during live production traffic.

---

Production Metrics at PayPal

PayPal’s real-world deployment illustrates the efficiency of this two-tier scaling strategy:

Dimension	Production Metric
Daily Request Volume	$> 100\text{ Billion requests/day}$
Storage Nodes	$\sim 200\text{ dedicated storage servers}$
Logical Shards	$1024\text{ fixed shards}$
Average Allocation	$\approx 5\text{ shards per storage server}$
Proxy Tier	Elastic, stateless pool scaled independently based on connection load

---

Key Architectural Takeaways

1. Do Not Overuse Consistent Hashing: Consistent hashing is designed for dynamic bucket counts. If partition boundaries can be fixed upfront, standard modular hashing (Hash(k) % N) is faster, simpler, and completely deterministic.
2. Decouple Partitioning from Placement: Use fixed logical partitions (shards) for keys, and use consistent hashing only to map those logical partitions to physical machines.
3. Separate Stateless Networking from Stateful Storage: By fronting storage engines with a stateless proxy layer that holds persistent backhaul connections, you insulate your core database from microservice connection spikes and TLS negotiation overhead.
4. Keep Data Mobility Units Granular: Breaking shards down into micro-shards allows continuous, low-overhead data migrations without triggering throughput degradation or latency spikes.