Essay #71: Data Partitioning Strategies
Partitioning plays a vital role in scaling a database beyond a certain scale of reads and writes. This essay takes a detailed look into the two common approaches to horizontally partition the data.
In this quick 3 minute read, we go through the following points
- What is data partitioning? - Why do we need to partition the data? - What if partitioning is skewed? - Data Partitioning Strategies - Range-based Partitioning - Hash-based Partitioning
What is data partitioning? A database is partitioned when we split it, logically or physically, into mutually exclusive segments. Each partition of the database is a subset that can operate as a smaller independent database on its own. Our primary goal with partitioning is to spread the data across multiple nodes, each responsible for only a fraction of the data allowing us to dodge the limitations with vertical scaling.
What if partitioning is skewed? Partitioning does help in handling the scale only when the load spreads uniformly. Partitions are skewed when few (hot) partitions are responsible for bulk data or query load. If the partitioning is skewed, the entire architecture will be less effective on performance and cost.
Range-based Partitioning One of the most popular ways of partitioning data is by assigning a continuous range of data to each partition, making each partition responsible for the assigned fragment. Every partition, thus, knows its boundaries, making it deterministic to find the partition given the partition key.
When Range-based partitioning fails? A classic use-case where range-based partitioning fails is when we range-partition the time-series data on timestamp. For example, we create per-day partitions of data coming in from thousands of IoT sensors.
Hash-based Partitioning Another popular approach for horizontal partitioning is by hashing the partitioned attribute and determining the partition that will own the record. Each partition owns a set of hashes. We hash the partitioned attribute when a record needs to be inserted or looked up. A partition that owns the hash will own and store the record.
When Hash-based partitioning fails? It is difficult to perform a range query on the data. Since the data is unordered and scattered across all partitions, we will have to visit all the partitions, making the entire process inefficient to perform a range query on key.
Here’s the detailed essay: https://lnkd.in/gxcJjReC.
This is the 15th essay in #DistributedSystems series, you can find other 71 essays at arpitbhayani.me/blogs
You can also subscribe to my free newsletter and get these essays delivered straight to your inbox: arpitbhayani.me/newsletter. 2800+ engineers have been reading my newsletter.