Building Highly Available Load Balancers with Anycast: Google’s 2010 Architecture

Designing distributed systems that remain resilient in the face of machine failures, regional fiber cuts, power outages, and catastrophic service crashes requires robust traffic management. In 2010, Google engineers published a paper titled “Anycast as a Load Balancing Feature”, outlining how they delivered load balancing and multi-site failover as an internal platform service.

While infrastructure tooling has evolved significantly since 2010, the architectural decisions, failure-mode analyses, and separation of concerns documented in this paper illustrate the fundamental networking mechanics used to design global-scale high availability.

---

1. The Multi-Tier Problem Statement

To understand why Anycast was chosen, we must first examine the natural progression of building load-balanced systems and identify where traditional approaches fall short.

flowchart TD
    A[Step 1: Multiple Service Replicas] --> B[Step 2: Active-Active / HA Pair Load Balancer]
    B --> C[Failure Mode: Entire Backend Fleet Crashes]
    C --> D[Mitigation: Proxy / Redirect to Secondary Site]
    D --> E[Failure Mode: Entire Site or LB Pair Collapses]
    E --> F[Solution: Automatic Regional Failover via Anycast]

Level 1: Service Replicas behind a Single Load Balancer

A standard on-premise service architecture places multiple service replicas (e.g., authentication, payments, profile APIs) in a server room behind a load balancer to distribute inbound traffic.

Level 2: High Availability (HA) Load Balancer Pairs

A single load balancer constitutes a single point of failure (SPOF). To mitigate this, load balancers are deployed in High Availability (HA) pairs (e.g., active-active or active-standby). If one load balancer machine fails, the second machine handles incoming traffic.

The Critical Edge Cases

Even with HA load balancing pairs, two disaster scenarios break system availability:

1. Total Backend Failure: A bad configuration push, runtime bug, or dependency crash brings down all service replicas in a server room simultaneously, while the local load balancers remain healthy.
2. Complete Site / Ingress Failure: A localized physical disaster (flooding, power loss, cut fiber uplinks) destroys both the load balancers and the backend servers in that site.

When this happens, traffic must be redirected to the nearest healthy secondary location without human intervention, complex cross-region proxies, or user-facing downtime.

---

2. Why DNS Failover Fails at Scale

A common way to reroute traffic between distinct geographical sites is DNS-based failover (e.g., updating CNAME or A records to point away from an unhealthy region). However, DNS-based failover introduces severe operational constraints:

• TTL Propagation Delays: Client-side DNS resolvers, ISPs, and operating systems frequently cache DNS entries beyond their Time-To-Live (TTL). Even if a health monitor detects an outage within seconds, users remain routed to a dead IP address until their local resolver TTL expires.
• Small TTL Overhead: Dropping TTLs to extremely small values (e.g., a few seconds) generates immense query traffic on DNS infrastructure, increases end-to-end request latencies due to frequent name resolution, and can still be ignored by misbehaving intermediate recursive resolvers.
• Distributed State Tracking: If a service operates in hundreds of edge locations or sites, deploying a centralized system that continuously health-checks every site and updates DNS records becomes a single point of failure and an engineering bottleneck.

---

3. Anycast Fundamentals and Limitations

What is Anycast?

In standard Unicast routing, each IP address belongs to exactly one host on the internet or private network.

In Anycast routing, multiple distinct hosts advertise the exact same IP address across network switches and routers using routing protocols such as BGP (Border Gateway Protocol).

When a client sends a packet to an Anycast IP:

1. Network routers evaluate the shortest path to that IP based on topological distance, network congestion, link status, and routing metrics.
2. The packet is automatically delivered to the topologically nearest healthy instance advertising that IP.

flowchart LR
    User[Client / User] --> Router{Internet / Core Router}
    Router -- Topologically Nearest --> SiteA[Site A: Load Balancer - IP 10.0.0.1]
    Router -. Longer Metric / Backup .-> SiteB[Site B: Load Balancer - IP 10.0.0.1]

The Limitation: Anycast Has No Application Awareness

Anycast operates strictly at the routing layer (Layer 3/Layer 4). Routers have no visibility into:

• Whether backend application containers or API servers are actually healthy.
• Application-level crash loops, database timeouts, or HTTP 500 status rates.

If an Anycast node advertises an IP, routers will direct traffic to it—even if every backend instance behind that node is dead. Routing protocol health checks only observe whether the router interface itself is reachable, not whether the upstream application is functional.

---

4. Google’s Solution: Decoupling Routing from Health via Anycast

Google solved this dilemma by combining Anycast routing with local health monitoring to create Anycast Failover as a Platform Service.

Instead of exposing every individual backend application host directly to Anycast (which would cause massive route churn across BGP tables if single services flickered), Google placed Load Balancers as Anycast Peers.

flowchart TD
    Client([Client Traffic]) --> VIP[Anycast Virtual IP: Shared Across Regions]
    
    subgraph SiteA [Site A - Primary]
        LBA[HA Load Balancer Pair] --> AppA1[Service Replica 1]
        LBA --> AppA2[Service Replica 2]
        LDirA[ldirectord Health Check] -.->|Monitors Backends| LBA
    end
    
    subgraph SiteB [Site B - Secondary Failover]
        LBB[HA Load Balancer Pair] --> AppB1[Service Replica 1]
        LBB --> AppB2[Service Replica 2]
        LDirB[ldirectord Health Check] -.->|Monitors Backends| LBB
    end
    
    VIP --> SiteA
    VIP -. Automatically swings if Site A stops advertising .-> SiteB

Key Principles of the Architecture

1. Decoupled Responsibilities:
   • Anycast (BGP) handles macro-level, inter-site topological routing and immediate multi-site failover.
   • Load Balancers (ipvs / LVS) handle micro-level, intra-site health checks, L4 distribution, and process monitoring.
2. Dynamic Route Withdrawal:
   • If local health checkers detect that all local backend replicas of a service have failed, the local load balancer triggers a script to withdraw the BGP advertisement for that service’s Virtual IP (VIP).
   • Surrounding network routers instantly sense that the route has vanished, recompute the BGP topology within seconds, and reroute incoming traffic to the next topologically nearest site.
3. Elimination of Remote Proxies:
   • Traditional cross-site failover proxies traffic from Site A to Site B across internal backbone connections, incurring double proxy overhead and high latency.
   • Anycast failover causes the client’s packets to go directly from edge routers to Site B, eliminating intermediate proxies.

---

5. Software Stack and Implementation Details

Google constructed this system using proven, open-source Linux networking primitives and custom-built integration hooks:

Component	Role in Architecture
Heartbeat (Linux-HA)	Maintains high availability between the two local load-balancer machines in a pair, ensuring if one server hardware node dies, the standby immediately claims the VIP.
IPVS (IP Virtual Server)	Transport-layer (Layer 4) load-balancing engine implemented in the Linux kernel (part of Linux Virtual Server / LVS), handling high-throughput TCP/UDP packet switching.
ldirectord	Linux Director Daemon. Polls application endpoints via HTTP/TCP checks to monitor backend replica health and modifies the IPVS routing table accordingly.
Quagga / BGP Daemons	Routing software suite that allows the Linux host to act as a BGP peer, announcing and withdrawing /32 host routes to upstream network routers.
ipconfig / iproute2	Interfaces with the Linux network stack to activate, bind, or tear down dummy interfaces containing the VIP.

The Custom Failover Mechanism

Google modified ldirectord with a custom fallback command:

1. ldirectord continuously performs health checks on the backend pool for each Virtual IP.
2. When backend instances fail individually, ldirectord instructs ipvs to drop those specific instances from the local pool.
3. If the last remaining local backend replica fails:
   • ldirectord triggers the custom fallback command.
   • The command tears down the local VIP interface or signals Quagga to withdraw the BGP /32 route advertisement.
   • Upstream network switches immediately stop routing traffic for that VIP to this server room.

Network Isolation and Security

To prevent rogue machines or misconfigured servers from advertising arbitrary IP addresses and hijacking corporate traffic:

• Dedicated subnets were reserved strictly for Anycast VIPs.
• Routers were locked down using Access Control Lists (ACLs) that only accepted /32 host route announcements from designated load balancer source addresses within specific subnets.

---

6. Recovery Times and Failure Modes

The real test of any failover system is its behavior across different failure boundaries:

Scenario A: Clean Service Outage

• Trigger: A deployment crashes all backend service replicas cleanly.
• Detection & Route Withdrawal: ldirectord detects the failed checks and drops the BGP peering route announcement in < 1 second.
• BGP Propagation: Core network routers re-converge in < 1 second.
• Total User-Perceived Outage: Health check interval (e.g., a few seconds) + ~1 second.

Scenario B: Catastrophic Hardware or Power Severing

• Trigger: A sudden power failure or fiber severed at an entire data center facility.
• Detection: Because the machine is hard-down, it cannot cleanly withdraw routes.
• BGP Dead Timer: Core routers must wait for the BGP Dead Timer (configured to 30 seconds) to expire without receiving Keepalive messages.
• Total Recovery Time: ~30 seconds + network route propagation delay (< 1 second).

---

7. Architectural Takeaways

1. Layered Separation of Concerns: Do not force Layer 3/4 network protocols to know about HTTP response codes, and do not force application processes to manage BGP routes. Use local orchestrators (ldirectord) as bridges that translate application health into route availability.
2. Load Balancers as BGP Aggregators: Having load balancers act as Anycast peers—rather than running BGP on every individual application container or VM—insulates the core network routing table from transient application restarts and thrashing.
3. Anycast Simplifies Developer Workflows: By packaging Anycast into a central load-balancing platform, service teams simply register their service under a Virtual IP. The platform automates topological routing, intra-site balancing, and zero-touch disaster recovery without requiring developers to master low-level networking.