Dissecting the GitHub MySQL Master Failover Outage

When a primary database fails over in a high-throughput system, operations must execute with zero room for error. During a routine planned maintenance window, GitHub attempted to migrate write traffic from an active MySQL primary (master) to a newly promoted standby primary. Within seconds of cutover, the new primary suffered a fatal process crash. Although traffic was immediately routed back to the original primary, that brief window introduced serious data divergence challenges and initiated a five-hour recovery process.

Here is an architectural dissection of why companies schedule database maintenance windows, how the failover failed, how data divergence occurs in split-second windows, and how MySQL binary logs (binlog) and replica reconstructions are used to recover consistent state.

---

Anatomy of an Incident: Reading the Symptoms

GitHub’s public incident report contained a telling observation:

“For a period of approximately 5 hours, users may have observed delays before data written to the affected database cluster were visible in the web interface and API.”

Notice the terminology: the issue was not data loss or complete write unavailability, but a delay in visibility.

In standard scaled architectures, database traffic is split between roles:

• Writes are handled by a single primary node (master) to preserve write consistency.
• Reads are offloaded to multiple read replicas to absorb high query volume.
• Asynchronous (or semi-synchronous) replication streams write events from the primary to read replicas.

flowchart LR
    App[Application Tier] -->|Writes| Master[(Primary / Master Node)]
    App -->|Reads| Replica1[(Read Replica 1)]
    App -->|Reads| Replica2[(Read Replica 2)]
    Master -.->|Asynchronous Replication| Replica1
    Master -.->|Asynchronous Replication| Replica2

When users can commit changes (e.g., creating an issue, merging a PR) but cannot see those changes reflected immediately on the UI or via read APIs, the write path is operational, but the read path is experiencing massive replication lag or degraded replica capacity.

---

Why Do Databases Require Planned Maintenance?

High-availability systems strive for zero downtime, yet organizations periodically schedule maintenance windows where non-operational status or temporary read/write pauses are planned. In database administration, major operational reasons necessitate deliberate reboots or failovers:

1. Applying Security Patches: The database engine (such as MySQL or PostgreSQL) is software running on an OS. Kernel updates, vulnerability patches (CVE mitigations), and minor engine patches often require a clean process restart.
2. Major/Minor Version Upgrades: Upgrading to a newer database version for performance optimizations, query engine improvements, or bug fixes frequently requires physical binary replacement.
3. Parameter & Buffer Tuning: While many database variables can be adjusted dynamically (SET GLOBAL), critical architectural changes—such as modifying the innodb_buffer_pool_size (depending on version), changing default page sizes, or swapping underlying storage engine configurations—mandate a service restart to allocate new memory structures.
4. Underlying Hardware Replacements: Cloud instances or bare-metal hypervisors suffer from hardware degradation, drive wear, or hypervisor retirements, requiring instances to migrate to fresh physical hardware.
5. Preventative Reboots: Clearing long-running memory fragmentation or cleaning up stale socket connections can necessitate scheduled maintenance cycles.

The Standard Zero-Downtime Failover Flow

To minimize user disruption, engineering teams do not take down the primary node while in active service. Instead, they promote a standby node:

sequenceDiagram
    autonumber
    participant App as Application Router
    participant OldM as Old Primary (Node A)
    participant NewM as Target Primary (Node B)
    
    Note over OldM,NewM: Node B is replicating continuously from Node A
    App->>OldM: Write traffic directed to Node A
    Note over App,NewM: Cutover Initiation
    OldM->>OldM: Set read-only / flush writes
    Note over NewM: Catch up on remaining replication stream
    NewM->>NewM: Promote to Read-Write
    App->>NewM: Update routing config / switch traffic to Node B
    App->>NewM: Write traffic directed to Node B

Under normal circumstances, this cutover takes milliseconds to seconds, causing at most a momentary blip in client connection pools.

---

What Went Wrong: The 6-Second Divergence Window

During GitHub’s planned cutover, traffic was routed to the newly promoted MySQL primary. Immediately after the cutover, the unexpected occurred:

“We experienced a novel crash in the mysqld process on the newly promoted MySQL primary server.”

Whether triggered by an unexpected edge-case query, a memory corruption bug, or resource exhaustion under immediate connection pressure, the mysqld daemon crashed.

The Immediate Fallback

To restore write availability, the site reliability engineers immediately reversed the configuration, routing write traffic back to the original primary (Node A). While intuitive, this revealed an acute distributed systems challenge:

The crashed primary had already accepted writes for approximately 6 seconds.

flowchart TD
    subgraph T0 [Phase 1: Prior to Cutover]
        A1[Old Master: Active Writes] --> B1[New Master: Replicating]
    end

    subgraph T1 [Phase 2: Cutover (6 Seconds)]
        B2[New Master: Serving Writes for 6s] 
        B2 -.->|mysqld process crashes| Crash[CRASH]
    end

    subgraph T2 [Phase 3: Emergency Fallback]
        A3[Old Master: Promoted back to Active Writes]
        A3 --> Divergence[Data Divergence! Node A lacks the 6s of writes from Node B]
    end

The Data Divergence Problem

Because Node B accepted live traffic for 6 seconds:

• Users received 200 OK responses for commits, comments, issues, and status checks.
• Those records existed only on the crashed Node B’s disk storage.
• Once traffic reverted to Node A, new writes continued on Node A.
• Node A and Node B now diverged in state: Node A had writes that occurred before $T_0$ and after $T_{failback}$, but completely missed the writes between $T_0$ and $T_{crash}$.

If left unaddressed, users would find recently submitted data missing, and auto-incrementing primary keys or unique index constraints would collide, corrupting relational integrity.

---

Resolving Divergence with Binary Logs (Binlogs)

In MySQL, every state-modifying event (insert, update, delete, schema change) is sequentially recorded in a Write-Ahead Log called the Binary Log (binlog).

Every event in the binlog has a specific file and offset, known as the binlog coordinate (e.g., mysql-bin.000142:1073741824) or a Global Transaction Identifier (GTID).

Binlog Coordinate Reconciliation

When initiating a planned failover, engineers track the exact binlog coordinate where the cutover took place.

Node B (New Primary) Binlog:
+--------------------------------+------------------------------------+
| Replicated from Node A         | Live Writes on Node B (6 seconds) |
+--------------------------------+------------------------------------+
^                                ^                                    ^
Start of file                    Failover Coordinate                  Crash Coordinate

To repair the data divergence:

1. Isolate Node B: Keep the crashed instance isolated from incoming network traffic to prevent any further modifications upon process restart.
2. Recover the Crash Log: Boot MySQL on Node B in recovery mode (e.g., running InnoDB crash recovery via redologs) to ensure the binlog is cleanly closed.
3. Extract Delta Events: Read the binary log from Node B starting strictly from the Failover Coordinate up to the point of the crash.

   mysqlbinlog --start-position=<FAILOVER_POS> \
               --stop-position=<CRASH_POS> \
               mysql-bin.000045 > delta_writes.sql

4. Apply Delta to the Active Primary (Node A): Replay the extracted transactions into Node A. Depending on whether GTIDs or statement/row-based logging is configured, collisions must be handled carefully to avoid key conflicts with transactions committed after fallback.

This reconciliation ensures that no write that received a confirmation is dropped.

---

Why Did the Outage Last 5 Hours?

Extracting and replaying 6 seconds of transaction logs takes minutes. Why did the incident report state an impact of 5 hours?

GitHub’s post-incident breakdown explained:

“At this point, a restore of replicas from the new primary was initiated which took approximately 4 hours, with a further hour for cluster reconfiguration to enable full read capacity.”

The Cascading Replica Invalidation

When a primary crashes and data diverges, read replicas can no longer safely replicate from either node without risking corrupt downstream state. A primary failover that partially completes leaves read replicas pointing to stale coordinates, or pointing to a primary that has diverged.

To guarantee strict consistency, the infrastructure team had to:

1. Re-establish Node A as the definitive source of truth after merging the 6 seconds of missing writes.
2. Rebuild the Read Replicas: Given the terabytes of data across GitHub’s primary relational clusters, snapshotting, restoring, and bringing read replicas into sync is an I/O-bound operation. Transferring and mounting database snapshots took roughly 4 hours.
3. Cluster Reconfiguration & Traffic Warming: Re-attaching the rebuilt replicas to replication topologies, allowing them to replay accumulated binlogs to reach zero replication lag, and updating proxy/routing tiers (like Vitess, ProxySQL, or internal connection routers) took an additional 1 hour.

gantt
    title GitHub Incident Resolution Timeline (5 Hours)
    dateFormat  HH:mm
    axisFormat %H:%M

    section Primary Mitigation
    Traffic Failback & Delta Reconciliation :active, 00:00, 00:30
    section Replica Rebuild
    Restoring Replicas from Snapshot        :00:30, 04:30
    Cluster Reconfiguration & Catchup       :04:30, 05:30

During this 5-hour reconstruction window, write operations proceeded, but read capacity was heavily constrained. Remaining or warming replicas struggled to keep pace, causing read requests to hit lagging replicas. This created the observed delay between a user saving data and seeing it displayed on GitHub interfaces.

---

Key Architectural Takeaways

Concept	Takeaway
Split-Brain & Failback Risks	A fallback to an old primary after a failed promotion is not free. Any writes accepted by the failed primary create instant data divergence that must be programmatically reconciled.
Replication Coordinates Matter	Tracking binlog files, offsets, and utilizing GTIDs (Global Transaction Identifiers) is mandatory during planned maintenance to know the exact boundaries of divergence.
Failover Automation Limits	Automated failovers must account for edge-case crashes occurring immediately post-promotion before secondary state machines assume the cluster is healthy.
Replication Rebuild Latency	The bottleneck of database recovery at petabyte scale is rarely write resumption—it is the physical time required to transfer, inflate, and sync read replicas across distributed clusters.