When I joined Google, I worked on building a Remote Shuffle Service for Dataproc and this is still the most challenging project I led…
Here’s a quick gist about why the problem is interesting and why companies like Uber, LinkedIn, Alibaba, and Databricks have implemented the same. Uber has even open-sourced their implementation, while others have well-written articles and papers.
Shuffle is the most expensive operation for any distributed processing and it happens when the partitioned data is not ideal for the engine to proceed to the next steps of computation.
For example, consider you are building GitHub, and your data is partitioned by repository_id, but you want to join on user_id to get the total number of commits per user per day. This would require data to be re-partitioned on user_id while this query is executing.
Because multi-TB shuffle takes a lot of time to complete, there is a chance that mappers fail and crash. This would lead to the entire work (owned by the mapper) to be re-computed. The solution to this is to separate storage and compute reducing the amount of rework required in case of failures.
Implementation of something as fascinating as RSS required us to go deep into Spark Internals and find the right distributed file system optimized for certain access patterns.
If this problem statement amuses you, just search for Remote Shuffle Service on Google, and you will find a ton of resources around it, mostly papers and a couple of OSS projects. Start going through them and you will have an excellent time understanding the nuances of Big Data Analytics.