How to decide the optimal number of partitions for a Kafka topic? ⚡
One of the most important things to know about Kafka is that its parallelism is limited by the number of partitions on the topic, which effectively means if you have n partitions on a topic then you may have at max n consumers consuming from the topic.
I consider 4 factors while deciding the number of partitions
- Scalability (involves number crunching)
- Hardware resources and limitations
- Retention policy
- Global ordering
It was hard to describe number crunching here, so I wrote a detailed write-up about it on Duggup, so head over there and read my detailed answer where I also explained how to crunch the numbers and plan the capacity.
Read - https://lnkd.in/gUXq2UQ2
⚡ I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along.
arpitbhayani.me/courses