Kafka Best Practices 🏆

Table Of Contents

• Managing Partitions Count

Managing Partitions Count

Partitioning Sizing

Partition counts cannot be changed after topic creation, so care should be taken in choosing the right size. If the number of partitions are too little, then fewer brokers are involved in handling the topic and there is a lot of serialized processing happening. As result, there could be producer lag and consumer lag.

Some brokers and consumers may be overworked handling big partitions while others may be starved for work as they don't have any partitions assigned to them. On the other hand, if the number of partitions is too high it results in more broker resources like file handles and memory. There is also high replication overhead if the number of partitions are high.

Recommendations

• Plan partitions counts before creating topics (as it is not possible to change them later).
• Make sure that the partition count is greater than the expected number of consumers (Partition count >= Number of consumers in a group).
• Use as many brokers as possible in the cluster as both partition leaders and replicas.
• Brokers have limitations in how many replicas they can handle including leaders and followers. Keep the total number of replicas per broker < 1 000. If needed, increase the number of brokers.
• Messages are distributed among partitions based on keys. So the total number of possible unique values for the key should be greater than the number of partitions.
• Perform load testing to understand the performance characteristics of specific application based on :
  • Overall peak load
  • Partition lag