Kafka Topic Design

Good topic design saves months of pain later. Partition counts are hard to reduce, names spread across many teams, and a poor key choice creates slow, uneven consumers. Think of topic design like planning the lanes of a highway before construction: widening a road after traffic arrives costs far more than planning it well.

Choose Clear Topic Names

A topic name should reveal its owner, its data, and its purpose. Adopt one naming pattern across the company.

  <domain>.<entity>.<event>

  shop.orders.created
  shop.payments.completed
  logistics.shipments.delivered

Use lowercase letters, dots, and underscores. Skip spaces and mixed styles, because Kafka treats dots and underscores as similar in metrics, which can cause name collisions.

One Topic per Event Type or Many?

Two approaches dominate real projects.

Approach	Best When	Trade-off
One topic per event type	Consumers need only some events	More topics to manage
One topic for related events	Order of different events matters for one key	Consumers filter unwanted events

Place events in the same topic when their relative order matters. An order that is created, paid, and shipped needs those three events in sequence, so one topic keyed by order ID keeps them aligned.

Pick the Partition Count

Partitions set the ceiling for parallel reading. A topic with six partitions supports at most six active consumers in one group. Estimate the count with a simple formula.

  Partitions = Target throughput / Throughput of one partition

  Target throughput        :  120 MB per second
  One partition handles    :   20 MB per second
  Partitions needed        :  120 / 20 = 6

  Add headroom for growth  :  choose 9 or 12

Measure the speed of one partition with your own hardware and message size. Numbers vary widely between systems.

Cost of Too Many Partitions

• More open files on each broker
• Longer leader election after a failure
• Higher memory use in producers and consumers

Cost of Too Few Partitions

• Consumers cannot scale beyond the partition count
• One broker may carry too much traffic

A count between 6 and 30 suits most topics. Start modest and grow when measurements demand it.

Select the Replication Factor

A replication factor of 3 protects most production data. It survives the loss of one broker while keeping writes available when min.insync.replicas is 2. Development clusters accept a factor of 1 or 2 to save space.

Design Message Keys

The key decides the partition. Choose a key that has many distinct values and matches the grouping you need for ordering.

• Order ID keeps all events of one order together
• Customer ID keeps all activity of one customer together
• A country code creates a hot partition if one country dominates traffic

  Good key : order-8841   -->  spread evenly across partitions
  Weak key : "India"      -->  most traffic lands in one partition

Set Retention Early

Decide how long data must stay available. Payment records may need seven days for replay, while click events may need only one. Write the retention value in the topic creation script so that every environment matches.

Plan for Change

Add a version field to message schemas and register them in a Schema Registry. Adding partitions later changes the key-to-partition mapping, so ordering for existing keys can break. Create new topics with generous partition counts when strict ordering matters.

Design Checklist

• Name follows the company pattern
• Partition count matches measured throughput with room to grow
• Replication factor is 3 for production
• Key supports ordering and spreads load
• Retention and cleanup policy are set
• Schema and owner are documented