Time series databases are extremely good at compression, and most use an interesting trick called Delta of Delta compression. Here’s how it works…
In time series data, values are typically recorded at regular intervals. Instead of storing the full 64-bit timestamp for each entry, you store only the difference (delta) from the previous timestamp.
But here’s where it gets interesting: if your data comes in at consistent intervals, say every 10 seconds, even those deltas will be nearly identical. So instead of storing the delta itself, you store the delta of the delta.
For example, if timestamps are 10, 20, 30, 40…
- Deltas: 10, 10, 10, 10
- Delta of deltas: 0, 0, 0, 0
When the variance is low (which it almost always is for time series), you end up storing zeros or tiny numbers that need only 1-4 bits instead of 64. That’s a compression ratio of up to 64x just on timestamps alone.
This same principle applies to values, too. Telemetry data, hardware metrics, latencies, response times, sensor readings, etc rarely jump wildly between consecutive measurements. Low variance makes the delta of delta super efficient.
Fun fact: This approach was made famous by Facebook’s Gorilla paper, and now it is a standard technique in databases like InfluxDB, TimescaleDB, and Prometheus.
Hope this helps.