Dissecting GitHub's Outage: When a Rate Limiter Causes Downtime

Rate limiters are designed to safeguard distributed systems from overload, resource exhaustion, and denial-of-service attacks. However, when improperly configured on internal service-to-service boundaries, defensive mechanisms can become single points of failure.

This incident occurred at GitHub when an internal rate limiter on an upstream configuration service triggered elevated error rates across the platform for users enrolled in A/B experiments. This deep dive examines the architectural anatomy of the failure, why the rate limiter triggered, how GitHub mitigated it, and the fundamental distributed systems lessons we can apply to our own architectures.

---

1. Fundamentals: A/B and Multivariate Experimentation

Modern engineering organizations continuously test UI changes, backend routing, algorithms, and workflows using A/B testing and multivariate experimentation.

               +-----------------------+
               | Incoming User Traffic |
               +-----------+-----------+
                           |
             +-------------+-------------+
             | User Experiment Allocation |
             +-------------+-------------+
                           |
            +--------------+--------------+
            |                             |
            v                             v
   +------------------+          +------------------+
   | Control Group A  |          | Treatment Group B|
   | (Default UI/API) |          |  (New Variant)   |
   +------------------+          +------------------+

In a standard setup:

• Control Group ($A$): Users receive the existing baseline functionality.
• Treatment Group ($B$ or $N$ in multivariate tests): Users receive the modified variant (e.g., an altered button layout, a modified data ingestion pipeline, or a different API routing strategy).
• Instrumentation & Metrics: The system monitors key performance indicators (conversion, latency, click-through rate, error rate) to statistically validate whether the treatment outperforms the baseline before initiating a 100% rollout.

To manage this dynamically without redeploying code for every tweak, organizations rely on centralized experimentation and feature-flagging services.

---

2. GitHub’s Architecture & The Critical Flaw

GitHub was rolling out enhancements to better instrument UI experiments. This setup relied on two primary components:

1. Frontend Application Servers: Responsible for rendering web views and routing user requests.
2. Configuration Generator Service: An upstream service responsible for dynamically generating experimentation configuration files that dictate UI variations, user bucket allocations, and telemetry endpoints.

+-------------------------------------------------------------------------+
|                        The Problematic Flow                             |
+-------------------------------------------------------------------------+

   +-------------------------+
   | Rolling App Deployment  |
   +------------+------------+
                | Boots up fleet of app instances
                v
   +-------------------------+
   | Frontend Server Fleet   |
   +------------+------------+
                | 
                | 1. Synchronous HTTP request for dynamic config file
                v
   +-------------------------+
   | Rate Limiter (Throttled)|
   +------------+------------+
                |
                | 2. Request REJECTED (Rate limit exceeded: 429/Error)
                v
   +-------------------------+
   | Config Generator (Down) |
   +-------------------------+

The Failure Chain

According to GitHub’s official post-incident analysis:

“Changes to better instrument A/B experimentation for UI introduced an unknown dependency on the presence of a specific dynamically generated file that is served by a separate application.”

1. The Hidden Dependency: Frontend instances were modified to depend synchronously on a dynamically generated configuration file upon initialization/execution.
2. The Deployment Trigger: During a routine application deployment, multiple frontend application pods/servers were spun up simultaneously. Each instance queried the upstream configuration service to retrieve and generate this file.
3. The Thundering Herd: The sudden spike in retrieval requests exceeded the threshold configured on the upstream service’s internal rate limiter.
4. Cascading Drop-off: The rate limiter actively throttled requests from the frontend instances. Because generation was dynamic and computationally demanding, the service could not fulfill the remaining requests in time.
5. Application Errors: The frontend servers, unable to acquire the mandatory configuration file, lacked a fallback mechanism. As a result, users assigned to the experimental cohort received application errors across the platform.

---

3. Incident Mechanics: Why the Rate Limiter Failed the System

Rate limiters are meant to prevent server crashes by discarding traffic exceeding safe capacity. However, in this case, the rate limiter directly induced a client-facing outage due to three architectural discrepancies:

A. Synchronous Initialization on the Critical Path

Dynamic configuration retrieval was bound directly to runtime traffic processing or container startup without local caching or static fallbacks. When the upstream call failed, the dependent application crashed or threw uncaught exceptions when serving experiment cohorts.

B. Misaligned Internal Rate Limiting Thresholds

Internal rate limiters must be calibrated to withstand legitimate cluster-level surges, such as:

• Fleet-wide rolling deployments.
• Cold-start autoscale events.
• Cache invalidation flushes.

When internal rate limits are configured with the same strict, low-tolerance thresholds as public-facing edge APIs, a healthy deployment looks indistinguishable from a Distributed Denial of Service (DDoS) attack to the upstream service.

C. Dynamic Generation vs. Precomputed Assets

Generating configuration files dynamically on demand requires CPU and memory cycles for each request. When hundreds of frontend nodes query the same endpoint simultaneously during a deployment, dynamic generation adds severe latency and saturates worker threads, rapidly exhausting rate limit quotas.

---

4. Mitigation & Remediation

+-------------------------------------------------------------------------+
|                         The Remediated Flow                             |
+-------------------------------------------------------------------------+

   +---------------------------+
   | Upstream Config Generator |
   +-------------+-------------+
                 |
                 | 1. Precomputes & writes to internal cache asynchronously
                 v
   +---------------------------+
   | Internal Cache / CDN      |
   +-------------+-------------+
                 ^
                 | 2. High-throughput, static reads (O(1))
   +-------------+-------------+
   | Frontend Server Fleet     |
   +---------------------------+

Short-Term Mitigation: Rollback & Decoupling

GitHub disabled the runtime requirement for the dynamically generated configuration file. Frontend instances fell back to baseline behavior without relying on upstream dynamic assets, immediately restoring service health.

Long-Term Architectural Fix: Precomputed Caching

GitHub updated the design to ensure configuration for A/B and multivariate experiments is cached internally:

• Decoupled Generation: The configuration generator builds the required state asynchronously or on configuration update, rather than on every frontend deployment request.
• Cached Propagation: Assets are pushed to a high-throughput, low-latency internal cache (or replicated distributed key-value store), allowing frontend nodes to read configuration with minimal load on the generation service.

---

5. Architectural Takeaways

1. Eliminate Synchronous Dependencies on Auxiliary Services

A/B testing, feature flagging, and experimentation are auxiliary capabilities—they are not core transaction paths (like reading a repository or writing a commit).

• Principle of Non-Critical Degradation: If an experimentation service fails, the system must automatically fall back to the default (control) behavior without raising errors to the user.
• Fail Open, Not Closed: For telemetry and experimentation metadata, services should default to standard execution paths if configuration cannot be fetched.

2. Differentiate Internal vs. External Rate Limiting

Rate limiting external clients protects against malicious actors and traffic spikes. Internal rate limiters serve a different role: load shedding under degraded states.

Dimension	External Rate Limiter	Internal Service-to-Service Rate Limiter
Primary Threat	Abuse, scrapers, volumetric attacks	Cascading failure, thundering herds during deploys
Threshold Policy	Strict, low-burst allowances	High burst allowances to absorb deployment waves
Client Behavior	Return 429 Too Many Requests	Return cache headers, stale fallbacks, or backpressure cues
Identity	IP, User ID, API Key	Service Principal, Spiffe ID, Subsystem Tier

3. Implement Strict Service Tiering (Tier 1 vs. Tier 2/3)

Organizations should categorize services into rigorous availability tiers:

• Tier 1 (Core Path): Services required to serve essential user requests (e.g., Git authentication, repository rendering). A complete failure here implies platform downtime.
• Tier 2 (Degradable Features): Important but non-essential capabilities (e.g., notifications, contribution graphs).
• Tier 3 (Auxiliary/Internal): Analytics, experimentation management, dynamic configuration builders.

The Golden Rule of Tiering: A service of a higher tier must never have a hard, synchronous dependency on a service of a lower tier. If a Tier 1 service interacts with a Tier 3 service, the boundary must be asynchronous, cached, or protected by circuit breakers and localized fallbacks.