Hot shards are a pain, but why do they occur? ⚡
Hot shards occur when a specific node of a distributed database, becomes overloaded and has to work significantly harder than other parts to handle the workload; leading to slowdowns and bottlenecks. Here are some common reasons for it
When I used to manage a production ElasticSearch cluster on raw EC2 virtual machines, hot shards were a big pain; and here are the reasons that were the root cause 80% of the times
- we did not follow the best practices
- we wrote highly inefficient queries
- we chose inefficient routing (partitioning) key
- unexpected surges happened due to performance marketing
- massive stop-the-world GC pauses, accumulating the backlog
Key things we did to address this is to have high observability in place along with spending time knowing the internals and best practices. This looked counter-productive at first, but it turned out to be the best decision in the long run.
note: although I mentioned these points w.r.t ElasticSearch, they hold true for almost all distributed databases that exist because the underlying limitations of hardware and software remain the same.
⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff.
youtube.com/c/ArpitBhayani