Should you learn sharding and partitioning if we already have a bunch of managed databases? According to me, an absolute yes! but more than that, you should know how to implement them, a few resources I highly recommend
- Even when the database is managed, you can get a better performance out of existing managed MySQL and Postgres by partitioning the data, helping you improve UX and reduce cost.
Read this explanation on StackOverflow: https://lnkd.in/gPVC8emn
- Knowing how the data is laid out on the disk and nodes to which each partition is assigned, you can pick the right partitioning key ensuring you get the best performance even from the managed database without requiring to overprovision it.
Read: https://lnkd.in/gbJBVXFF
- The concept of data partitioning is very similar to split ownership which you would require on the compute side to understand how you would do distributed computing (and even single-node multi-threaded programs) such that you maximize the hardware utilization.
Watch: youtu.be/2PjlaUnrAMQ
- Knowing these is also essential when you work with Spark because even though there are managed Spark offerings, by partitioning your data well and understanding Physical Plan, you can bring down your Spark job execution time and in turn the cost.
Read about: Broadcast Nested Loop Join
- Knowing how to partition data without downtime, and move shards across without downtime is something that will help you do no-downtime database migrations (from one DB to another).
Watch: youtu.be/9iAJjtvBwyI
I believe these are the concepts that you should know in and out because now distributed computing and split ownership patterns are becoming quite common and Sharding and Partitioning form the crux of it, but you and others may have a different opinion on this.
In case, you want to understand Sharding and Partitioning from absolute scratch and the real difference between watch youtu.be/wXvljefXyEo.
ps: 8 seats left in my sys design feb cohort - arpitbhayani.me/course
It is a no-fluff and highly practical course that reignites your engineering curiosity and helps you become great at designing, implementing, and shipping scalable, fault-tolerant, and highly available systems at your workplace. In the course, you will
- learn the intuition behind designing, building, and implementing systems
- brainstorm, interact, and learn from the entire cohort and their experiences
- build prototypes & understand implementation details and operational challenges of running things in production
No random boxes, but rather extremely practical details around how systems are built, scaled, and shipped.