Database Management at Scale: Dissecting the GitHub MySQL Outage

When a primary database crashes in production, manual intervention is often too slow to meet strict service level agreements (SLAs). Modern infrastructure relies on specialized tooling to automatically detect failures, elect a candidate replica, reconfigure the topology, and route traffic seamlessly.

However, automated failover can backfire catastrophically when an underlying issue causes newly promoted primaries to crash repeatedly. This post explores how massive architectures manage MySQL clusters, detailing the roles of ProxySQL and Orchestrator, the anatomy of GitHub’s cascading MySQL outage, and the critical role of anti-flapping mechanisms in disaster recovery.

---

The Production Database Architecture

At high scale, application servers do not establish direct TCP connections to underlying database nodes. Doing so causes connection storms, architectural rigidity, and operational overhead during failovers. Instead, systems introduce an intermediate proxy and an out-of-band topology manager.

flowchart TD
    subgraph Application Layer
        A1[API Server 1]
        A2[API Server 2]
        A3[API Server N]
    end

    subgraph Routing Layer
        P[ProxySQL Cluster]
    end

    subgraph Topology Management
        O[GitHub Orchestrator]
    end

    subgraph MySQL Cluster
        M[(Primary / Writer)]
        R1[(Read Replica 1 - Small Frequent Reads)]
        R2[(Read Replica 2 - Heavy Analytics)]
    end

    A1 & A2 & A3 -->|MySQL Protocol| P
    P -->|Writes| M
    P -->|Fast Reads| R1
    P -->|Heavy Reads| R2
    M -.->|Async/Semi-Sync Replication| R1
    M -.->|Async/Semi-Sync Replication| R2
    O -->|Monitor & Topology Updates| M
    O -->|Monitor & Topology Updates| R1
    O -->|Monitor & Topology Updates| R2
    O -.->|Repoint Primary Host| P

This architecture relies on two critical tools:

1. ProxySQL: An in-path, high-performance proxy speaking the MySQL protocol.
2. Orchestrator: An out-of-band daemon responsible for discovery, health checking, and topology refactoring.

---

Deep Dive: The Role of ProxySQL

ProxySQL sits between client applications and backend MySQL servers. Clients treat ProxySQL as if it were a native MySQL instance, while ProxySQL abstracts away cluster management.

1. Connection Rationing and Pooling

MySQL handles connections via threads or thread pools. If thousands of API instances connect directly, the database spends excessive CPU cycles on thread context switching and connection state management.

• ProxySQL maintains a persistent pool of long-lived connections to backend databases.
• It multiplexes thousands of incoming application connections onto a strictly bounded set of backend database connections, protecting the database from saturation.

2. Intelligent Query Routing & Read/Write Splitting

ProxySQL can inspect SQL syntax and route queries based on configurable rules:

• Writes (INSERT, UPDATE, DELETE) are directed to the primary node.
• High-frequency, point-lookup SELECT queries are routed to standard read replicas.
• Resource-heavy analytics queries against massive tables are directed to isolated, dedicated reporting replicas.

Because the routing logic resides entirely in the proxy, backend topology changes do not require code changes or redeployments in the application layer.

3. Query Caching

ProxySQL can cache query result sets directly in memory. When identical, idempotent read queries arrive (e.g., SELECT * FROM users WHERE id = 10), ProxySQL returns the cached result without hitting MySQL, reducing read load and query latency.

4. Temporary Credential Management

Directly sharing long-lived database credentials with engineering teams is a security risk. ProxySQL allows administrators to generate short-lived, time-bound credentials (e.g., valid for 1 hour) for ad-hoc debugging without exposing or altering primary database users.

---

Deep Dive: GitHub Orchestrator

While ProxySQL manages the data path, Orchestrator manages the control plane. Orchestrator continuously discovers, monitors, and restructures MySQL topologies.

1. Topology Discovery & Visualization

Orchestrator periodically polls MySQL nodes to build a live dependency graph of the cluster. It monitors replication topology, replication lag, binary log coordinates, and cross-data-center replication structures across dozens of read replicas.

2. Automated Crash Detection and Failover

When a primary node becomes unreachable or halts:

1. Orchestrator confirms that the node is genuinely down (avoiding false positives from network blips).
2. It selects the most up-to-date replica (the one with the lowest replication lag and latest binary log position).
3. It promotes that replica to become the new primary writer.
4. It repoints the remaining read replicas to replicate from the newly promoted primary.
5. It notifies the infrastructure (e.g., updating ProxySQL routing tables) to send write traffic to the new host.

---

The Incident: What Happened at GitHub

A critical incident occurred involving automated database failover, ProxySQL, and Orchestrator.

The Sequence of Events

1. June 22 (Maintenance Window): The database infrastructure team deployed an updated version of ProxySQL and pushed supporting application changes.
2. June 29 (Incident Day - 1 Week Later): A primary MySQL node in one of GitHub’s main clusters crashed unexpectedly.
3. Automated Promotion: Orchestrator detected the primary failure and automatically promoted a healthy replica to become the new primary writer.
4. Immediate Secondary Crash: Within seconds of promotion, the newly appointed primary suffered severe CPU starvation and crashed.
5. Anti-Flapping Tripped: Orchestrator’s anti-flapping safeguards engaged, blocking subsequent automated promotions.
6. Manual Promotion Failure: Operations engineers manually intervened and promoted another replica to primary. It immediately hit 100% CPU starvation and crashed as well.
7. Mitigation: With every promoted node crashing under the workload, the team reverted the ProxySQL upgrade, rolled back the associated application changes, and manually recovered the topology. The cluster stabilized, ending a 2.5-hour write outage.

---

Understanding Cascading Failures and the Anti-Flapping Pattern

The Cascading Failure Loop

When a primary node crashes, the cluster loses capacity. If an underlying workload issue or memory/CPU leak triggered the initial crash, promoting a replacement node can trigger a catastrophic domino effect:

sequenceDiagram
    participant Load as Inbound Traffic
    participant P1 as Primary 1
    participant R1 as Replica 1 (Candidate)
    participant R2 as Replica 2
    participant O as Orchestrator

    Load->>P1: Normal + Pathological Traffic
    P1->>P1: CPU Starvation / Crash
    O->>R1: Automated Failover: Promote to Primary
    Load->>R1: Shift All Write Traffic to R1
    R1->>R1: CPU Starvation / Crash
    Note over O: Anti-Flapping Engages! (Auto-failover blocked)

If auto-failover continues unchecked, every replica in the cluster will be successively promoted, overwhelmed, and knocked offline. Within minutes, an entire read-and-write database cluster can be wiped out.

The Anti-Flapping Safeguard

To prevent cascading cluster destruction, Orchestrator incorporates an anti-flapping mechanism:

• Throttling Promotions: After an automatic failover occurs, Orchestrator enforces a cool-off window (typically 5–10 minutes) before allowing another automated promotion in the same cluster.
• Failing Closed: If the newly promoted primary crashes within that window, Orchestrator assumes the cluster is unstable and refuses to perform automated failovers.
• Enforcing Human Intervention: Traffic may fail, but existing read replicas remain operational to serve read queries. Operations engineers are forced into the loop to inspect the root cause before sacrificing the remaining nodes.

---

Critical Engineering Takeaways

1. Failures Can Have Long Latency Gaps

The ProxySQL upgrade occurred a full week before the cluster failure. Deployment-related bugs are not always immediately evident; they may depend on specific workload thresholds, query patterns, or periodic database maintenance routines.

2. Centralized Infrastructure Change Logs

Correlating an outage with an event that occurred days prior requires a single, queryable source of truth for all changes across the organization. Engineering teams should be able to view every configuration update, package bump, and rollout across dependencies during incident triage.

3. Maintain Post-Mortem Observability Artifacts

When nodes crash due to resource starvation, transient metrics may fail to capture the root cause. Deep debugging requires:

• MySQL Core Dumps: Memory snapshots captured at the moment of a process crash to inspect stack traces.
• ProxySQL Query Logs: Detailed access and error logs to identify abnormal query spikes or malformed queries.

4. Full Revert as the Primary Incident Triage Tool

When multiple manual and automated recovery attempts fail, attempting to debug code live in production prolongs the outage. A complete rollback—reverting the application code, proxy versions, and configuration files to the last known stable state—is the safest and fastest way to restore service.