Zero-copy writes make Kafka fast, but how exactly? ⚡
Kafka is a message broker, and it accepts messages from the network and writes to the disk, and vice versa. The traditional way of moving data from network to disk involves read and write system calls, which require data to be moved to and from user space to kernel space.
Kafka leverages sendfile system call which copies data from one file descriptor to another within the kernel. Kafka uses this to directly transfer data from the network socket to the file on disk, bypassing unnecessary copies.
If you are interested, just read the man page of sendfile system call. In most cases, whenever you see something extracting extreme performance a major chunk of it comes from leveraging the right system call.
ps: I used this zero copy while building Remote Shuffle Service for Apache Spark. It proved pivotal in getting a great performance while moving multi-tb data across machines.
⚡ Admissions for my System Design June 2024 cohort are open, if you are SDE-2 and above and looking to build a rock-solid intuition to design any and every system, check out
arpitbhayani.me/course