How Amazon Aurora Achieves Durability with Quorum Writes and Segmented Storage

This article delves into the critical aspect of durability in database systems, specifically exploring how Amazon Aurora, a cloud-native relational database, ensures data safety through its innovative quorum-based write model and segmented storage architecture. Building upon the concept of storage-compute separation, we examine the mechanisms that allow Aurora to withstand various failures, from individual disk corruptions to entire availability zone outages, while maintaining high availability and rapid recovery.

Understanding Durability in Distributed Systems

Durability, in its simplest form, means that once data is written and acknowledged, it can be reliably read back, even in the face of system failures. This seemingly straightforward concept becomes complex in distributed environments where data is written across multiple nodes over a network.

A key principle of durability is that instance lifetime is not equal to storage lifetime. This implies that even if a database instance goes down (due to graceful shutdown, scaling, reboots, or hardware failures), the data it managed must persist. In large-scale cloud environments, failures are not rare events but a continuous, low-level background noise. Transient network lags, temporary downtimes, machine reboots, permanent disk failures, node failures, rack failures, or even entire data center outages are all possibilities that a robust database service must account for.

The Quorum-Based Approach to Durability

To ensure durability, Amazon Aurora employs a quorum-based model, a fundamental concept in distributed systems for achieving consistency and fault tolerance.

What is a Quorum?

A quorum refers to a decentralized set of nodes that must agree on a particular state or operation. In the context of durability, it ensures that data is safely stored and retrievable even if some nodes fail.

The core idea is to maintain V copies of replicated data. For any operation (read or write) to be considered successful, a certain number of nodes must participate:

• Read Quorum (VR): A read operation is successful only when VR nodes respond with the requested data. The system then typically takes the most common or latest value based on a resolution strategy.
• Write Quorum (VW): A write operation is considered complete only when it has been successfully applied to VW nodes.

Critical Quorum Conditions

Two fundamental conditions must be met for a quorum system to guarantee consistency and durability:

1. Overlap Condition: VR + VW > V This condition ensures that there is always at least one node with the newest version of the data. When a read is issued, its quorum (VR) will necessarily overlap with the quorum of the latest successful write (VW), guaranteeing that the read operation will encounter at least one node that holds the most recent data.

2. Majority Write Condition: VW > V / 2 This condition is crucial to prevent conflicting writes and ensure that every successful write is aware of the most recent prior write. If VW is not greater than half of V, it’s possible for two conflicting writes to be acknowledged by disjoint sets of nodes, leading to data inconsistency.

   Example:

   • Assume V = 10 nodes.
   • If VW = 2 (not > V/2 which is 5).
   • Write 1 goes to nodes A and B, gets acknowledged.
   • Conflicting Write 2 goes to nodes I and J, gets acknowledged.
   • Both writes are considered successful, but they are conflicting, and there’s no overlap between {A, B} and {I, J}. When a read occurs, it’s ambiguous which value is the “latest” or correct one.
   • By enforcing VW > V/2, any two write quorums will always have at least one overlapping node, ensuring that subsequent writes are aware of previous ones and preventing split-brain scenarios.

Inadequacy of 2x3 Quorum at Scale

A common quorum configuration often discussed is the “2x3 quorum,” where V=3, VR=2, and VW=2. This setup satisfies both VR + VW > V (2+2 > 3) and VW > V/2 (2 > 3/2). While simple and effective for smaller deployments, it is inadequate at Amazon’s scale due to the high probability of multiple concurrent failures.

Consider the following scenarios for a 2x3 quorum:

1. All 3 nodes in one Availability Zone (AZ): If that single AZ goes down (a common occurrence due to network failure, power outage, or natural calamity), the entire database becomes unavailable.
2. Two nodes in AZ1, one node in AZ2: If AZ1 goes down, only one node remains available in AZ2. This is insufficient to meet the VW=2 or VR=2 quorum, rendering the database unavailable.
3. One node in AZ1, one in AZ2, one in AZ3: If one AZ goes down, two nodes remain. While this initially seems sufficient, in a large fleet, the probability of one of the remaining two nodes failing (e.g., disk failure, machine reboot) while the first AZ is still down is significantly high. This “double fault” scenario would again lead to quorum loss and database unavailability.

At Amazon’s scale, the probability of a double fault is sufficiently high that a simple 2x3 quorum is not robust enough.

Amazon Aurora’s Durability Strategy: A+1 Failure Tolerance

To address the challenges of large-scale failures, Amazon Aurora is designed to tolerate an “A+1” failure scenario: the failure of an entire Availability Zone plus one additional node failure in another AZ, while still maintaining database availability for reads, and often for writes.

Aurora achieves this by:

• Six Copies of Data: Instead of 3, Aurora maintains six copies of every data segment.
• Distributed Across AZs: These six copies are strategically distributed across three Availability Zones, with two copies stored in each AZ.

Aurora’s Quorum Configuration

With V=6 copies, Aurora uses the following quorum parameters:

• Write Quorum (VW) = 4
• Read Quorum (VR) = 3

Let’s verify the conditions:

• VR + VW > V: 3 + 4 = 7, which is > 6. (Condition met)
• VW > V/2: 4, which is > 6/2 = 3. (Condition met)

Failure Tolerance Analysis with Aurora’s Quorum

1. One Availability Zone (AZ) Failure:

   • If one AZ goes down, two nodes become unavailable.
   • Four nodes remain available (two from each of the other two AZs).
   • Writes: VW=4 can still be met, as all four remaining nodes can acknowledge the write.
   • Reads: VR=3 can still be met from the four remaining nodes.
   • Result: Both reads and writes remain available.

2. One AZ Failure + One Additional Node Failure (A+1 Fault):

   • If one AZ goes down (2 nodes unavailable), and one additional node from the remaining four also fails.
   • Three nodes remain available.
   • Writes: VW=4 cannot be met (only 3 nodes available). Writes become unavailable.
   • Reads: VR=3 can still be met from the three remaining nodes.
   • Result: The database remains available for reads, even though writes are temporarily unavailable. This is a significant improvement over a 2x3 quorum, where a double fault would typically lead to complete unavailability.

This robust configuration ensures a high degree of durability and availability even under severe failure conditions.

Segmented Storage for Enhanced Durability and Operational Simplicity

Beyond the quorum model, Amazon Aurora employs a segmented storage architecture to further enhance durability, reduce blast radius, and improve mean time to repair (MTTR).

10GB Segments as the Unit of Storage

• Aurora’s storage volume (which can be up to 64 TB) is logically divided into fixed-size 10GB segments.
• Crucially, a segment is not equivalent to a physical disk. Instead, these 10GB segments are the fundamental units of storage and replication.
• Each 10GB segment is replicated six times across three Availability Zones, forming a “protection group.”

Rapid Repair and Reduced Blast Radius

The segmented approach offers several key benefits:

• Fast Recovery: If a 10GB segment becomes corrupted, unavailable, or the node hosting it fails, Aurora can quickly repair it. With five other copies available, a new sixth copy can be created by streaming data from other nodes over a 10Gbps link, typically completing the repair within 10 seconds. This dramatically shrinks the MTTR.
• Reduced Blast Radius: Failures are localized to individual 10GB segments rather than entire disks or larger volumes. If a disk fails, only the specific 10GB segments on that disk need to be repaired, not the entire disk’s worth of data.
• High Fault Tolerance for Segments: For a specific 10GB segment to lose its quorum (and thus its durability), a highly unlikely scenario must occur: two nodes holding the same segment must fail, plus a complete failure of the AZ holding that same segment. This is considered sufficiently unlikely even at Amazon’s scale.

Operational Simplicity and Self-Healing Infrastructure

The segmented storage design also brings significant operational advantages:

• Self-Healing: If a segment becomes “hot” (experiencing high I/O) or is identified as “bad” (corrupted or unavailable), the system can automatically mark it as such. The quorum mechanism then kicks in, initiating the repair process by replicating the segment to a new location, often resolving the issue within seconds.
• Disk Management: If an entire disk becomes hot or fails, all segments residing on that disk can be marked as bad. The distributed quorum system will then automatically re-replicate all affected segments to healthy storage, effectively self-healing the infrastructure.
• Rolling Releases and Patching: This architecture facilitates seamless rolling updates. Administrators can mark a node or volume as “bad” for maintenance (e.g., patching, security upgrades). The quorum system automatically handles the replication of its segments to other nodes, allowing for zero-downtime updates as segments are repaired in parallel.

This pattern of partitioning data into smaller, manageable units is a common strategy in large-scale distributed systems for efficient heat management and operational resilience.

Conclusion

Amazon Aurora’s approach to durability is a testament to sophisticated distributed systems design. By combining a robust quorum-based write model with six-way replication across Availability Zones, it achieves an impressive A+1 failure tolerance. Furthermore, its innovative segmented storage architecture, with 10GB segments as the unit of protection and repair, significantly reduces recovery times and simplifies operations, making Aurora a highly durable and resilient database solution for critical workloads.