DeepSeek open-sourced their high-performance distributed file system optimized for AI training and inference workloads 🤯
The file system is called 3FS. They also released performance numbers, and they are pretty impressive.
Its Sorting Performance on the popular GraySort Benchmark outperforms Spark + HDFS and even AWS EMR by 2x minimum 🤯 On a large cluster, it also offers a higher read throughput of 6.6 TiB/s, where AWS S3 hits only ~5 TiB/s.
What blew my mind was its KVCache read throughput numbers. 3FS offers a 40 GiB/s peak throughput, while NVIDIA offers 10–20 GiB/s, which is a 4x improvement while being more cost-efficient than even DRAMs.
I am digging deeper right away because after a long time, some breakthrough came in AI that I hold the background to understand better :-)
DeepSeek is changing the game and how!
The link to the source code is in the comments.