Kafka Partition Count Calculator

A Kafka topic is split into ordered partition logs. That split determines how much producer and consumer work can happen in parallel, how leaders and replicas are distributed across brokers, and where record order is preserved. A count that is too small can cap throughput or leave consumers idle. A count that is much larger than needed increases metadata, file, replication, recovery, and rebalance work.

Three demands can set the minimum count:

• Write throughput needs enough leaders to accept the peak producer rate without exceeding a measured per-partition benchmark.
• Read throughput needs enough partitions for the busiest consumer group to keep up with its own processing rate.
• Consumer parallelism needs at least as many partitions as active consumers in one group if every consumer is expected to receive work.

The busiest group is sized independently. Ten full-rate consumer groups increase broker read traffic, but they do not require ten times as many partitions for assignment. Within an ordinary consumer group, total consumer threads are limited by the partitions available to that group.

Keys add a separate design constraint. Records assigned by key keep their order within one partition. Increasing a live topic's count changes the partition mapping used for future keyed records under common hash partitioning, so records for the same key may land in a different partition after expansion. Kafka can add partitions, but it cannot reduce a topic's partition count in place.

Good planning begins with benchmarks from the actual serialization, compression, acknowledgement, storage, broker, and consumer path. A growth buffer protects against forecast error; upward rounding can improve broker or consumer balance. Neither should replace cluster-specific limits or a review of replica placement, controller load, recovery time, and key-ordering behavior.

How to Use This Tool:

Size one topic with peak workload measurements and the busiest consumer group that must process the stream in real time.

1. Choose a Workload preset as a starting point, then switch Sizing basis to measured throughput or event rate and average serialized size. Both paths produce write and read targets in MiB/s.
2. Enter benchmarked Producer capacity per partition and Consumer capacity per partition, then set Peak consumers in one group. Do not add independent consumer groups together for this parallelism floor.
3. Set eligible brokers, replication factor, minimum in-sync replicas, and keying posture. Replication factor must not exceed the broker count, and minimum in-sync replicas must not exceed replication factor.
4. Choose a Growth buffer and an upward rounding rule. Use Capacity drivers to see whether write rate, read rate, consumer count, buffering, or rounding forced the final count.
5. Open the advanced inputs when planning an existing topic or enforcing local guardrails. Compare Current partitions, leader density, remaining cluster budget, and preferred topic maximum before using the generated topic command.

Interpreting Results:

Recommended partition count is the buffered requirement after the chosen upward rounding rule. Trace it back through Capacity drivers; a large jump caused only by rounding may call for a different balance rule rather than more topic shards.

• Benchmark again if the modeled write or read rate per partition approaches the capacity value used as an input. The result inherits every limitation of that benchmark.
• Use Broker footprint to review leader and replica density. Replication changes broker work and placement count, not consumer-group parallelism.
• Stop before expanding a keyed topic until key remapping, per-key order, consumer discovery, and rollout sequencing have been reviewed.
• If the current topic has more partitions than the recommendation, keep the current count or plan a replacement and migration. The lower modeled number is not an in-place shrink instruction.
• Treat any cluster-budget or preferred-maximum warning as a local approval failure even when throughput math passes.

Technical Details:

Partition sizing takes the maximum of independent write, read, and consumer floors. It then applies one growth factor and one upward rounding rule. This order matters: rounding each driver separately or adding the drivers would produce a different and unsupported count.

Formula Core:

When event-rate mode is selected, messages per second and average KiB per message are converted to MiB/s before the same capacity equations are used.

The buffered count is rounded up to the eligible broker count, peak consumer count, their least common multiple, the next power of two, or no additional multiple. Broker/consumer least-common-multiple rounding can create a large step when those counts share few factors.

Rule Core:

Total replica placements equal recommended partitions × replication factor. Average leaders per broker divide partitions by eligible brokers. Average replica placements per broker divide total replicas by brokers. Broker read traffic multiplies the busiest-group target by the entered number of full-rate consumer groups before dividing across brokers.

Limitations:

This is a capacity-planning model, not a broker benchmark or placement simulator. It assumes the supplied per-partition rates are sustainable and treats broker distribution as an average. It does not model skewed keys, compression ratio, batch size, acknowledgement latency, storage saturation, rack placement, retention, tiered storage, controller limits, consumer lag recovery, or failures during reassignment. Validate the chosen count in a representative cluster and apply local operating limits.

Worked Examples:

References:

• Basic Kafka Operations, Apache Kafka 4.3 Documentation.
• KafkaConsumer API, Apache Kafka 4.1.2 Documentation.
• How to configure Filebeat output to Kafka, Simplified Guide.