In the world where you have managed databases, should you learn sharding and partitioning? ⚡
According to me, the answer is an absolute yes! you should know these concepts and more importantly how to implement them, here’s why.
- Even when the database is managed, but you can get a better performance out of existing managed MySQL and Postgres by partitioning the data, helping you improve UX and reduce cost.
Read: https://lnkd.in/gPVC8emn
- Knowing how the data is laid out on the disk and nodes to which each partition is assigned, you can pick the right partitioning key ensuring you get the best performance even from the managed database without requiring to overprovision it.
Read: https://lnkd.in/gbJBVXFF
- The concept of data partitioning is very similar to split ownership which you would require on the compute side to understand how you would do distributed computing (and even single-node multi-threaded programs) such that you maximize the hardware utilization.
Watch: youtu.be/2PjlaUnrAMQ
- Knowing these are also essential when you work with Spark because even though there are managed Spark offerings, by partitioning your data well and understanding Physical Plan, you can bring down your Spark job execution time and in turn the cost.
Read about: Broadcast Nested Loop Join
- Knowing how to partition data without downtime, and move shards across without downtime is something that will help you do no-downtime database migrations (from one DB to another).
Watch: youtu.be/9iAJjtvBwyI
I totally believe these are the concepts that you should know in and out because now distributed computing and split ownership patterns are becoming quite common and Sharding and Partitioning form the crux of it, but you and others may have a different opinion on this.
In case, you want to understand Sharding and Partitioning from absolute scratch and the real difference between watch youtu.be/wXvljefXyEo.