Stripe’s Zero-Downtime, Consistent Data Migration Strategy at Scale

Data migration is a ubiquitous challenge in software development, especially as systems evolve and requirements change. It involves transforming data from one model to another—whether adding/removing columns, changing data types, or restructuring entities. While seemingly straightforward, performing a massive data migration without downtime and ensuring absolute data consistency is a complex feat. This document dissects Stripe’s battle-tested approach to online data migrations at scale, drawing insights from their real-world implementation.

Why Data Models Evolve

Data models typically evolve for two primary reasons:

1. Cleaner Abstractions: As systems grow, initial data models can become messy or difficult to extend. Evolution allows for refactoring into cleaner, more maintainable, and extensible structures.
2. Evolving Requirements & Complex Features: New business requirements or the development of more complex features often necessitate changes to how data is stored and represented.

For SQL databases, this might involve running an ALTER TABLE query. However, at scale, the real challenge lies in executing these changes while guaranteeing consistency (data never enters an incorrect state) and availability (the system remains operational).

The Challenge of Large-Scale Online Migration

Consider a hypothetical scenario, inspired by Stripe’s own numbers: migrating 100 million subscription objects from one schema to another. If each object takes just 1 second to read, transform, and write sequentially, the total migration time would be 100 million seconds, approximately 3 years. This is an unacceptable duration for any online system. The core requirements are non-negotiable:

• Zero Downtime: The system must remain available throughout the migration.
• Data Accuracy & Consistency: No data loss or corruption is permissible.

Stripe’s Core Strategy: Dual Writing

Stripe employs a standard yet nuanced approach: dual writing. This technique allows for a gradual transition by writing data to both the old and new schemas simultaneously, ensuring that both remain up-to-date during the migration process.

High-Level Four-Step Process

From a 10,000-foot view, the dual-writing strategy involves four key steps:

1. Dual Write: New data is written to both the old and new tables/collections.
2. Change Read Paths: Application code is updated to read data from the new table/collection.
3. Change Write Paths: Application code is updated to write data exclusively to the new table/collection.
4. Delete Old Data: Once confidence is established, the old table/collection is decommissioned and deleted.

While these steps seem simple, the devil is in the details, especially when dealing with millions of records and strict consistency requirements.

Stripe’s Concrete Example: Subscription Model Evolution

Stripe’s specific migration involved evolving their customer-subscription model. Initially, they assumed a one-to-one relationship: one customer, one subscription. Their MongoDB-based (or DogDB, built on MongoDB) schema reflected this with a nested subscription object within the customer object:

// Old Schema
{