Dissecting GitHub’s Integer Overflow Outage: Post-Incident Protocols and Mitigation vs. Resolution

When a large-scale system goes down, an engineering organization shifts into high-stress incident mode. The top priority is immediate: stop the bleeding and restore traffic. However, once services are marked green on the status dashboard, the job is far from over.

A common engineering failure is conflating incident mitigation with incident resolution. Restoring service availability is merely mitigation; full resolution requires handling data inconsistencies, purging poisoned caches, retrofitting alerting gaps, and hardening systems so that specific failure classes never recur.

In this teardown, we analyze the GitHub outage caused by an integer overflow in scope tokens, examine how foreign key mismatches slip past automated checks, and outline the four essential practices required after an outage is mitigated.

---

The Incident: An INT32 Overflow on Scope Tokens

During this incident, users experienced broad failures across GitHub Actions, GitHub Pages, GitHub APIs, and Git operations such as git push and git pull.

Root Cause Analysis

The outage stemmed from a database write failure: a foreign key column referencing scope tokens hit the maximum limit of a 32-bit integer (MAX_INT32).

-- Simplified Representation of the Problem
CREATE TABLE tokens (
    id BIGINT PRIMARY KEY, -- 64-bit integer (safe up to ~9.22 x 10^18)
    token_hash VARCHAR(255)
);

CREATE TABLE scoped_token_references (
    id BIGINT PRIMARY KEY,
    token_id INT NOT NULL, -- Sneaky bug: 32-bit signed int (overflows at ~2.14 billion!)
    scope VARCHAR(50),
    FOREIGN KEY (token_id) REFERENCES tokens(id)
);

In standard relational databases (such as MySQL or PostgreSQL):

• A signed 32-bit integer caps at $2^{31} - 1 = 2,147,483,647$ (~2.14 billion).
• An unsigned 32-bit integer caps at $2^{32} - 1 = 4,294,967,295$ (~4.29 billion).

For low-throughput platforms, 2 to 4 billion records feels practically infinite. However, for right-heavy services operating at global scale, integer exhaustion occurs rapidly.

                   +------------------------------------+
                   |        High-Frequency Writes       |
                   |  (GitHub Actions, Pages, Push/Pull)|
                   +-----------------+------------------+
                                     |
                                     v
                    +---------------------------------+
                    | Scope Token Generation Service  |
                    +----------------+----------------+
                                     |
                     Fails to insert | token_id > 2,147,483,647
                                     v
              +----------------------------------------------+
              |  `scoped_token_references` Database Table    |
              |  Column: `token_id INT` (Signed 32-bit)      |
              |  ERROR: Integer value out of range for column|
              +----------------------------------------------+

Why Scope Tokens Exhausted First

GitHub Actions, GitHub Pages, and authenticated Git operations (push/pull) are exceptionally write-heavy:

• Every commit, push, lint check, and CI/CD workflow run issues and authenticates short-lived scope tokens.
• Every Git CLI operation against the remote API issues or inspects token references.
• At GitHub’s volume, hundreds of millions of token authorizations are requested, burning through 32-bit ID spaces at an alarming velocity.

The Sneaky Failure: Primary Key vs. Foreign Key

Most database reliability teams monitor primary keys for auto-increment thresholds. GitHub had already upgraded the primary key (id) of the parent tokens table to an INT64 (BIGINT).

However, a related table held a foreign key reference (token_id) that remained typed as INT32. As soon as the primary key passed 2,147,483,647, downstream inserts containing that valid 64-bit ID into the 32-bit foreign key column threw fatal database errors. Because writes failed unconditionally, all dependent operations ground to a halt.

Immediate Mitigation: Online Schema Migration

The only remediation was updating the foreign key column type from INT32 to INT64 (BIGINT). Modifying a column type on a table with billions of rows locks tables or requires long-running asynchronous online schema migrations (e.g., using tools like gh-ost or pt-online-schema-change). Until the schema migration completed across all replicas and primary databases, the system remained impaired.

---

Mitigation vs. Resolution

A critical operational principle to uphold is:

Mitigation $\neq$ Resolution

• Mitigation means bringing the system back online to an operational state (e.g., flipping a feature flag, completing the schema migration, or scaling replicas). Traffic flows, and errors drop.
• Resolution means handling the technical fallout of the outage: repairing inconsistent states, evicting corrupt or poisoned cache lines, filling observability blind spots, and eliminating the underlying failure mode across the organization.

+-----------------------+
|      Active Outage    |  Writes failing, errors spiking
+-----------+-----------+
            |
            v  (Apply hotfix, schema change, restart, failover)
+-----------------------+
|  Mitigation Complete  |  System accepts traffic; errors return to baseline
+-----------+-----------+
            |
            |  1. Reconcile partial/orphaned writes
            |  2. Invalidate stale/invalid cache entries
            v  3. Audit alerting and schema linters
+-----------------------+
|  Resolution Complete  |  Data integrity verified; safeguards prevent recurrence
+-----------------------+

---

The Post-Outage Playbook: 4 Critical Steps

Once GitHub finished the schema migration and restored database writes, the engineering team executed systematic cleanup procedures. Any engineering team managing mission-critical systems should follow these four operational steps.

1. Detect and Remediate Data Inconsistencies

When a core data tier crashes or rejects writes midway through upstream operations, partial state changes occur across boundary services.

Consider an atomic pattern across separate boundaries:

Step 1: Write state to Database A (Succeeded)
Step 2: Write state to Database B (Failed due to INT overflow)

If services lack distributed transaction coordination (such as 2-Phase Commit or Sagas with compensating transactions), your system enters an inconsistent state.

Following an outage:

• Inspect tables and write-ahead logs for orphaned records created during the failure window.
• Scan for partial updates where Step 1 succeeded but Step 2 failed.
• Run data reconciliation scripts to re-sync or cancel incomplete operations before user traffic interacts with corrupted entities.

2. Invalidate Poisoned and Stale Cache Layers

During write failures, cache coherence protocols inevitably break:

• Partial writes cached: An application might write a generated token to Redis or Memcached before hitting the database failure, leaving an invalid record in the cache.
• Failed eviction calls: Cache invalidation webhooks or events sent over message queues may have been dropped or dead-lettered during system saturation.

In GitHub’s incident, while the database had rejected token references, the cache layer still held records for partially generated, invalid tokens. GitHub engineers had to manually comb through and evict those token records from the cache layer so clients were not rejected with invalid authentication payloads.

Best practice: Always trace whether your cache contains keys that were written without corresponding durable database commitments, and implement selective or bulk cache invalidation pipelines.

3. Audit Alerting Strategies and Linter Blind Spots

GitHub already had static database linters and threshold alerting designed to catch impending integer overflows. How did this outage bypass those controls?

The Alerting Gap: The linting rules were configured to inspect primary keys (PRIMARY KEY AUTO_INCREMENT). The foreign key column in question predated the introduction of those automated linting rules, leaving it unmonitored.

To prevent similar misses:

• Never limit integer capacity alerting exclusively to primary keys; run monitoring against all integer columns (both primary and foreign keys).
• Alert well before limits are breached (e.g., at 60%, 75%, and 85% capacity), providing adequate lead time to schedule multi-day schema migrations on massive tables.
• Perform comprehensive audits of alerting systems after an incident to identify other assets that slipped past legacy automation.

+-------------------------------------------------------------------------+
|                      Integer Capacity Audit Rule                        |
+-------------------------------------------------------------------------+
|  SELECT table_name, column_name, data_type                              |
|  FROM information_schema.columns                                        |
|  WHERE data_type IN ('int', 'integer', 'smallint')                      |
|  AND table_schema NOT IN ('information_schema', 'sys', 'performance_schema');|
+-------------------------------------------------------------------------+
      | 
      v Scan values dynamically against MAX_INT to evaluate current headroom

4. Implement Permanent Class-Level Preventions

Fixing a single column prevents that exact table from breaking again, but it does not prevent the same class of failure from happening elsewhere in the fleet.

To achieve true closure:

• Audit all columns fleet-wide: GitHub initiated an exhaustive manual and automated audit of every single INT32 column across all database clusters.
• Enforce migration gates in CI/CD: Any new schema pull request introducing a numeric ID or reference column should automatically fail linting if defined as INT32, enforcing BIGINT (INT64) by default.
• Improve automation: Upgrade schema analysis tools to inspect production statistics periodically and flag tables running out of range long before manual intervention is needed.

---

Key Takeaways

1. Default to BIGINT for Relational IDs: Storage savings of 4 bytes per row between an INT32 and INT64 are rarely worth the operational risk of a global outage on high-throughput platforms.
2. Foreign Keys are Silent Killers: Upgrading a primary key to 64-bit while leaving its foreign key references as 32-bit creates a delayed time bomb.
3. Mitigation is Half the Job: An incident only concludes once inconsistent records are repaired, invalid caches are cleared, and blind spots in linting and telemetry are patched.
4. Never Fail the Same Way Twice: Post-mortems must focus on eliminating an entire class of bugs across the organization rather than merely patching a single isolated column or service.