Cloudflare published a blog on how they serve Kimi K2.6 and GLM 5.2 at scale, and the memory math behind it is worth digging into.
For these models, it is usually not the model weights that fill up GPU memory first. It is the KV cache.
For Kimi K2.6, Cloudflare stores the cache in FP8 (8-bit) precision instead of the default BF16 (16-bit). That halves its size, and it takes the max context held in memory from about 686,000 tokens to 1.37 million tokens. Pretty solid.
Here is the interesting part…
At any fixed concurrency, BF16 is a few percent faster per token (~5-9%), since the FP8 kernel has to convert its 8-bit cached values to 16-bit precision before it can do the attention math.
BF16 skips that conversion entirely. But BF16 stores each cached value at double the size, so it runs out of cache room at 32 concurrent requests and cannot admit a 33rd.
By using FP8 instead, Cloudflare keeps admitting requests up to 64, and that is where the 41% higher peak throughput comes from, by fitting more requests onto the same GPU at once.
For GLM 5.2, they go further and compress the weights themselves, from FP8 down to INT4. This makes per-GPU memory go from about 88 GB to 52 GB.
We might feel that cutting the bit width in half, twice over, sounds like it should cost something in quality. It actually does not.
FP8 KV cache and BF16 land within a point of each other across benchmarks. INT4 weights stay within 0.8 points of FP8 across every benchmark they ran.
So this is the trade Cloudflare is making: smaller storage everywhere, with benchmarks confirming nothing is lost, in exchange for far more concurrent requests on the same GPU.