Google just tested the largest Kubernetes cluster with 130,000 nodes 🤯, and all this because AI is pushing the boundaries!
The GKE setup was able to scale to 130,000 Pods in just 3 minutes and 40 seconds. During this test, there was a sustained Pod throughput of 1,000 Pods per second.
What’s driving this need for scale? A single NVIDIA GB200 GPU requires 2,700W of power. When you’re running tens of thousands of these chips, you’re looking at hundreds of megawatts for a single cluster. Looks like we’re shifting from being limited by chip supply to being limited by electrical power.
To handle this, Google rebuilt core Kubernetes systems. They used a Spanner-based storage backend that processed 13,000 queries per second required for lease updates. The cluster managed over 1 million objects (k8s metadata) while keeping the API server (central hub of k8s) latency stable.
Key design decisions:
- Snapshottable API server cache
- Consistent reads from the API server cache instead of hitting the DB
- Spanner-based storage for handling massive metadata query load
- GCS FUSE with parallel downloads and caching for efficient data access
Yes. It was all about caching :) and caching things well.
I have linked the blog from Google below. Give it a read. It’s quite fun.