Today, in my sys design cohort, we were discussing LSM trees, and an interesting question came up…
Some context - in LSM trees, writes first go to an in-memory buffer. This buffer is later flushed to L0 as SSTables. These SSTables are then compacted into L1, which is range partitioned. Also, SSTables are immutable, which means no in-place updates and require complete rewrites.
So the natural question is: if L1 can hold the entire dataset, why do we even need L2? This comes down to controlling write amplification so it does not spiral out of control. Let me explain …
L1 is intentionally kept small. That keeps compaction from L0 into L1 relatively cheap. If L1 were large enough to hold the full dataset, every compaction from L0 would force a small batch of new data to be merged with a very large L1 SSTables (even if there is one key change in the range).
That is, rewriting huge amounts of data again and again, which quickly becomes super expensive.
For example, imagine L0 has just 10 MB of new data. If those keys are spread across the key space and overlap with a 10 GB L1, the system may need to read and rewrite most of that 10 GB just to integrate those 10 MB. That is where things break down.
Keeping L1 small avoids this problem.
With a small L1, say around 256 MB. L0 data might overlap with only a few L1 files (smaller files, having that broad range), so maybe about 30 MB gets rewritten, i.e., 3x amplification, which is manageable.
With a large L1, say 100 GB, new keys are likely spread across many parts of L1, compaction touches a large portion of L1 (larger files and smaller ranges per file), so 10 MB in can lead to gigabytes rewritten, which is not sustainable.
So instead of putting everything into L1, most of the data is pushed down into L2 and lower levels. These levels are much larger, often about 10x bigger than the previous one. Data moves down only when upper levels fill up.
This means most data eventually becomes cold and sits in L2 or beyond, where it is rarely rewritten.