If protobuf is already a compact binary format, should we even bother compressing it further with Snappy? Here’s what Uber’s M3 actually does…
Uber’s M3 ingests over a billion datapoints per second (a pretty insane scale), and it actually ingests data over HTTP/1.1 as Snappy-compressed protobuf.
Protobuf is efficient, yes, but telemetry data has a unique characteristic: it’s highly repetitive. You see the same metric names, similar tag sets, repeated hostnames, and consistent field structures across thousands of data points. Given how redundant it is, this data can be compressed extremely well, even with a fast algorithm like Snappy.
M3 uses Snappy-compressed protobuf over HTTP for its Prometheus remote write endpoint. The reason is simple - at a massive scale, you need compression that won’t become your bottleneck.
- Snappy compresses and decompresses extremely fast
- it prioritizes speed over compression ratio
- the CPU overhead on both client and collector stays minimal
- telemetry clients shouldn’t burn cycles on compression when they should be doing actual work
Fun fact: at scale, everything is a bottleneck. You try to save every bit and every CPU cycle. All you need to do is understand the use case really well, observe patterns, and pick what suits you best.