Load Balancers Are Not Magic: Dissecting the Atlassian Outage

In 2019, Atlassian experienced a significant outage shortly after rolling out a new change. This incident, which self-recovered within five minutes without any manual intervention, provided crucial insights into the often-misunderstood behavior of load balancers. The core lesson, as Atlassian’s own blog post titled, is that “Load Balancers are not Magic.” This article dissects the outage to understand the underlying mechanisms and extract valuable learnings for designing and operating scalable systems.

The Atlassian Outage: A Postmortem

When the outage occurred, Atlassian’s engineering team observed a perplexing situation:

Initial Observations

• Service Metrics Normal: API servers appeared healthy. CPU utilization, memory usage, and other application-level metrics were within normal ranges.
• No Error Logs: There were no exceptions or error logs generated by the application servers.
• Load Balancer Request Dip: Crucially, the number of requests observed at the load balancer level dropped to zero during the outage duration. This indicated that requests were not even reaching the application servers.

Given that Atlassian’s infrastructure runs on AWS and utilizes AWS-managed Elastic Load Balancers (ELBs), they contacted AWS for deeper insights.

AWS Insights: The OOM Error

AWS’s internal dashboards revealed a critical piece of information: newly created ELB servers (the internal machines powering the abstract ELB entity) were dying due to Out Of Memory (OOM) errors.

This highlighted a fundamental aspect of load balancers: they are not single, monolithic machines. An ELB is an abstraction behind which multiple load balancer servers operate. These internal servers horizontally scale to handle incoming request loads, just like application servers.

Load Balancer Scaling Characteristics

Load balancers are designed to scale, but their scaling isn’t instantaneous:

• Horizontal Scaling: When the incoming request load increases, the ELB adds more internal load balancer servers to distribute the traffic.
• Warm-up Time: A critical nuance is the time it takes for a newly added load balancer server to spin up and become ready to serve requests. This “warm-up” period typically ranges from 1 to 7 minutes. During this time, the new server is not contributing to handling the load.

Surges vs. Spikes

This warm-up time is why load balancers behave differently under various load patterns:

• Surges: Gradual increases in request volume (e.g., over several minutes or hours) can be handled effectively. The load balancer has enough time to scale out its internal servers incrementally.
• Spikes: Sudden, extreme increases in request volume (e.g., flash sales, sudden marketing pushes, or major live events like a cricket match) are problematic. If the spike is too steep and too fast, the load balancer may not be able to provision and warm up new servers quickly enough to absorb the sudden influx of requests. This can lead to request drops or OOM errors on existing servers if they are overwhelmed before new capacity comes online.

Interestingly, Atlassian did not observe a sudden spike in overall traffic leading up to the outage. This deepened the mystery.

Unraveling the Root Cause: Atlassian’s Deployment Strategy

The key to understanding the outage lay in Atlassian’s deployment strategy at the time.

Original Blue-Green Deployment Architecture

Atlassian employed a form of blue-green deployment, but with a critical variation:

1. New Infrastructure Creation: For a new service version, they would spin up an entirely new set of API servers.
2. New ELB Creation: Crucially, they also created a brand new Elastic Load Balancer for the new infrastructure.
3. DNS Flip: Once the new infrastructure (API servers and ELB) was ready, they would update the DNS entry (e.g., svc1.atlassian.com) to point from the old ELB’s IP address to the new ELB’s IP address. This would redirect all incoming traffic to the new version.

The Critical Flaw: The Unwarmed ELB

The problem arose from the combination of creating a new ELB for each deployment and the ELB’s warm-up characteristics:

• Initial State of New ELB: When a new ELB is created, AWS typically provisions it with a minimal number of internal load balancer servers (e.g., one or two). It is “cold” or “unwarmed.”
• Sudden Load Transfer: When the DNS entry was flipped, the entire production load (e.g., 100,000 requests per second) was instantaneously redirected to this newly created, unwarmed ELB.
• Capacity Mismatch: If the unwarmed ELB could only handle, say, 20,000 requests per second with its initial servers, the remaining 80,000 requests per second would be dropped or cause the existing ELB servers to become overwhelmed and crash (OOM errors), as observed by AWS.
• Scaling Delay: The ELB would then detect the high load and begin scaling out by adding more internal servers. However, this process takes 1 to 7 minutes for each new server to become operational.
• The Outage Window: During this 5-minute window, while the ELB was actively scaling and warming up to meet the demand, a significant portion of incoming requests failed, leading to the observed outage. Once enough ELB servers were warmed up and ready, the system automatically recovered.

This explains why Atlassian saw no application errors (requests weren’t reaching the app servers) and no overall traffic spike (the total traffic remained constant, but its destination shifted abruptly).

The Fix

The solution was straightforward:

• Reusing Existing Load Balancers: Instead of creating a new ELB for every deployment, Atlassian modified their strategy to reuse the existing, warmed-up load balancer.
• Rolling Deployments: They transitioned from a full blue-green deployment (which included replacing the ELB) to a rolling deployment model where only the API servers were replaced or updated, keeping the load balancer constant.

By doing so, they ensured that the load balancer was always adequately scaled and warmed to handle the production traffic, eliminating the “cold start” issue.

Key Takeaways and Best Practices

1. Load Balancers Are Not Magic: Understand that load balancers, especially managed services like AWS ELB, have internal mechanisms and limitations. They are not infinitely scalable or instantly reactive.
2. Warm-up Times Matter: Be aware of the time it takes for load balancers to scale and for new internal servers to become operational (typically 1-7 minutes).
3. Pre-warming for Spikes: If you anticipate sudden, extreme spikes in traffic (e.g., marketing campaigns, flash sales), contact your cloud provider (e.g., AWS) to pre-warm your load balancers. This ensures they are provisioned with sufficient capacity in advance.
4. Understand Your Deployment Strategy: Carefully consider how your deployment strategy interacts with your infrastructure components, especially load balancers. A full blue-green deployment that replaces the load balancer can introduce cold start issues if not managed correctly.
5. Monitor Load Balancer Metrics: Beyond application metrics, closely monitor load balancer-specific metrics (e.g., request count, healthy host count, latency at the LB) to detect issues early.
6. Design for Scalability and Resilience: Factor in the scaling characteristics of all components, including load balancers, when designing your system architecture to avoid unexpected outages during load shifts or scaling events.