How Stripe Achieves Zero-Downtime, Consistent Data Migrations at Scale

Data migration is a fundamental challenge in software engineering, especially when evolving data models for large-scale systems. While seemingly straightforward, performing a massive online migration without downtime or data inconsistency is a complex endeavor. This article dissects Stripe’s approach to tackling this challenge, drawing insights from their real-world experience migrating millions of subscription records. The strategies discussed are applicable to any organization facing similar data evolution needs.

Why Data Models Evolve

Data models are rarely static; they evolve for two primary reasons:

1. Cleaner Abstractions: As systems grow, initial data models might become messy or difficult to extend. Engineers often seek to refactor and introduce cleaner, more modular abstractions to improve maintainability and future extensibility.
2. Evolving Requirements & Features: Business requirements change, and new, more complex features are built on existing foundations. This often necessitates changes to how data is structured and stored to support new functionalities efficiently.

For SQL databases, such changes might involve ALTER TABLE queries. However, the real complexity arises when these operations must be performed on a live production system, guaranteeing both data consistency (data never enters an incorrect state) and system availability (the system remains operational without interruption).

The Challenge of Massive Online Migrations

Consider a scenario like Stripe’s, where 100 million subscription objects need to be migrated from one schema to another. If each object transformation (read, transform, write) takes just 1 second, a sequential migration would require 100 million seconds, approximately 3 years. Such a prolonged migration is unacceptable for a critical financial system, highlighting the need for highly efficient, distributed, and online migration strategies.

The core requirements for such a migration are:

• Zero Downtime: The system must remain fully operational throughout the migration.
• Data Consistency: No data should be lost, corrupted, or become inconsistent.
• High Accuracy: All data must be correctly transformed and moved.
• Scalability: The process must handle massive volumes of data efficiently.

Stripe’s Core Strategy: Dual Writing

Stripe, like many other large-scale systems, employs a standard yet nuanced approach: Dual Writing. This technique involves writing data to both the old and new schemas/tables simultaneously during a transition period. From a 10,000-foot view, the process typically involves four steps:

1. Dual Write: New data changes are written to both the old and new data stores.
2. Change Read Paths: Application logic is updated to read data from the new data store.
3. Change Write Paths: Application logic is updated to write exclusively to the new data store, ceasing writes to the old.
4. Delete Old Data: Once confidence in the new system is established, the old data store is decommissioned.

While these steps seem simple, the devil is in the details, especially when dealing with the scale and criticality of Stripe’s operations.

Deep Dive into Stripe’s Subscription Migration

Stripe faced a specific migration challenge: initially, their data model assumed one customer would have one subscription, nested within the customer object (using MongoDB/DocDB).

Initial Schema (Conceptual):

{
  "customer_id": "cust_123",
  "name": "John Doe",
  "subscription": {
    "subscription_id": "sub_abc",
    "plan": "premium",
    "status": "active"
  }
}

As requirements evolved, customers could have multiple active subscriptions. The solution was to move subscriptions into a separate collection, establishing a one-to-many relationship.

Evolved Schema (Conceptual): customers collection:

{
  "customer_id": "cust_123",
  "name": "John Doe"
}

subscriptions collection:

{
  "subscription_id": "sub_abc",
  "customer_id": "cust_123",
  "plan": "premium",
  "status": "active"
}

Here’s how Stripe executed this migration with meticulous attention to detail:

Step 1: Implementing Dual Writes and Backfilling Historical Data

The first phase involves ensuring all new data changes are reflected in both the old and new data models, while also migrating existing historical data.

Asynchronous Dual Writes

To avoid doubling transactional latency, Stripe implemented asynchronous dual writes. New write operations would first commit to the old collection and then asynchronously propagate to the new collection. This ensures that the primary transactional path remains fast, while the new collection gradually receives up-to-date data.

Distributed Backfill of Old Data

Migrating 100 million existing records sequentially on a live production database is not feasible due to performance impact and the sheer time required. Stripe’s solution involved:

1. Database Snapshot: Taking a snapshot of the old production database. This isolates the backfill process from live traffic.
2. Distributed Map-Reduce: Utilizing a distributed system (e.g., MapReduce workers) to process the snapshot.
   • Workers iterate through the old snapshot in a distributed fashion.
   • They identify records needing migration.
   • They perform the necessary data transformation.
   • They write the transformed data to the new collection.

This approach ensures that the backfill operation does not affect the performance of the live production database, as all read load for the backfill is directed to the snapshot.

Data Verification Post-Backfill

After the distributed backfill job completes, a crucial verification step is performed to ensure no data was missed during the migration. This involves iterating and checking that all relevant old data has been correctly moved and transformed into the new collection, ensuring a “proper, fully stable movement.”

Step 2: Shifting Read Paths with Consistency Guarantees

Once new writes are dual-written and historical data is backfilled, the next step is to direct application reads to the new data model.

Flipping Read Paths

This involves a code change to update all application read paths to query the new collection instead of the old one.

Shadow Mode and Data Comparison

Blindly trusting the new read path is risky. To guarantee consistency and prevent serving inconsistent data to users, Stripe implemented a shadow mode:

• For a period, reads are performed from both the old and new collections.
• The results from both reads are compared.
• If any mismatch is detected, an alert is immediately raised.

This shadow mode is paramount for ensuring a seamless transition and maintaining user trust. It acts as a critical safety net, verifying that the data in the new system is identical to what would have been read from the old system.

Step 3: Transitioning Write Paths and Ensuring Rollback Capability

With reads confidently directed to the new system, the final step before decommissioning the old data is to transition write paths.

Asynchronous Writes to Old for Rollback

Initially, writes went to the old collection and asynchronously to the new. Now, this is flipped: writes go to the new collection and are asynchronously propagated to the old collection.

Why continue writing to the old collection? This is a critical Plan B strategy. In the event of an unforeseen issue or a need to revert the migration, having the old collection up-to-date (or at least recent) allows for a seamless rollback to the previous state of the system. This foresight is essential for large-scale, high-stakes migrations.

Codebase Challenges and Best Practices

Changing write paths often involves modifying thousands of lines of code across a large codebase. Stripe emphasizes:

• Modularization and Abstraction: Structuring the codebase to make such changes as simple and localized as possible.
• Simplicity: Ensuring the changes are easy for engineers to understand and implement, instilling high confidence.
• Rollback Mechanism: Explicitly designing and testing a clear way to revert the changes if necessary.

The risk of data inconsistency for a financial system like Stripe is unacceptable. Therefore, this phase requires rigorous verification and a slow, deliberate rollout, despite the perceived slowness.

Step 4: Safely Decommissioning Old Data

The final step is to remove the old data, but this must be done with extreme caution.

Thorough Verification of No Dependencies

Before deleting the old data, it is imperative to verify that:

• All code changes have been deployed and are stable.
• All configuration changes have been applied.
• Absolutely 100% of transactional traffic is no longer directed to the old system.
• There are no “stray leaked connections” or forgotten configuration strings pointing to the old database.

Premature deletion could lead to severe outages if any part of the system still relies on the old data store.

Data Deletion

Only after comprehensive verification and absolute certainty that the old data is no longer needed or accessed by any part of the system, the old data can be safely deleted.

Key Takeaways

Stripe’s approach to online data migration highlights that while the high-level steps (dual write, change reads, change writes, delete) appear simple, the real complexity lies in the meticulous implementation details and the robust mechanisms required to guarantee:

• Zero Downtime: Achieved through asynchronous operations and careful sequencing.
• Data Consistency: Ensured by shadow mode comparisons, rigorous verification steps, and rollback capabilities.
• Scalability: Handled by distributed processing on database snapshots.
• Resilience: Built-in Plan B (asynchronous writes to old for rollback) for unforeseen issues.

For any critical system, especially those dealing with financial data, prioritizing data correctness and integrity throughout the migration process is paramount. This often means embracing a slower, more deliberate approach with multiple layers of verification and fallback mechanisms.