Zero-copy writes make Kafka fast, but how exactly?
Kafka accepts messages from the network and writes to the disk, and vice versa. The traditional way of moving data from the network to the disk involves read and write system calls, which require data to be moved to and from user space to kernel space.
Kafka leverages the sendfile system call, which copies data from one file descriptor to another within the kernel. Kafka uses this to directly transfer data from the page cache to the network socket, bypassing unnecessary copies. This is the zero-copy optimization people keep talking about.
The performance difference is massive. The traditional approach involves 4 copies
- Disk → Kernel buffer (DMA)
- Kernel buffer → User space
- User space → Socket buffer, and
- Socket buffer → NIC (DMA).
With sendfile, we only have 2 copies:
- Disk → Kernel buffer (DMA), and
- Kernel buffer → NIC (DMA) via descriptor passing.
Kafka also relies heavily on the OS page cache. The data typically lives in memory (PageCache), and sendfile transfers directly from there to the network, making it even faster.
One caveat: sendfile doesn’t work with TLS/SSL because encryption needs to happen in user space, and it requires data to be unmodified.
If you are interested, just read the man page of the sendfile system call. In most cases, whenever you see something extracting extreme performance, a major chunk of it comes from leveraging the right system call.
By the way, I used this zero-copy while building the Remote Shuffle Service for Apache Spark. It proved critical in getting a great performance while moving multi-terabyte data across machines.
Hope this helps.