Kafka Consumer Rebalancing
A consumer group shares the partitions of a topic among its members. Rebalancing is the process that redistributes those partitions whenever the group changes. Imagine four waiters serving ten tables. When one waiter leaves or a new one arrives, the manager reassigns tables so that everyone carries a fair load. Kafka performs the same reshuffle automatically.
What Triggers a Rebalance
- A new consumer joins the group
- A consumer leaves or shuts down cleanly
- A consumer crashes or stops sending heartbeats
- A consumer takes too long between poll calls
- The topic gains new partitions
Rebalance in Pictures
The topic payments has four partitions. Two consumers read from it at the start.
BEFORE AFTER Consumer C joins
Consumer A : P0, P1 Consumer A : P0, P1
Consumer B : P2, P3 Consumer B : P2
Consumer C : P3
Partition P3 moves from B to C. Consumer B stops reading P3 and Consumer C begins at the last committed offset, so no message gets skipped.
The Group Coordinator
One broker acts as the group coordinator for each group. It tracks members, receives heartbeats, and starts a rebalance when membership changes. A group leader, which is one of the consumers, calculates the new assignment and sends it to the coordinator for distribution.
Eager and Cooperative Rebalancing
| Mode | Behavior | Impact |
|---|---|---|
| Eager | Every consumer releases all partitions, then receives new ones | Full pause for the whole group |
| Cooperative | Only partitions that must move get released | Most consumers keep reading |
Select the cooperative strategy with this setting:
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
The sticky logic keeps partitions with their current owners whenever possible, which lowers the amount of movement.
Settings That Control Stability
| Setting | Role | Common Value |
|---|---|---|
| session.timeout.ms | Time without heartbeat before a consumer counts as dead | 45000 |
| heartbeat.interval.ms | Delay between heartbeats | 3000 |
| max.poll.interval.ms | Longest gap allowed between poll calls | 300000 |
| max.poll.records | Messages returned by one poll | 500 |
Keep the heartbeat interval at roughly one third of the session timeout. A short session timeout detects crashes quickly but triggers false alarms during brief network hiccups.
Why Rebalances Hurt
During an eager rebalance the group stops consuming. Lag grows, latency rises, and in-memory state may be lost. Frequent rebalances, sometimes called a rebalance storm, can stall an application for minutes.
Common Causes of Rebalance Storms
- Slow message processing that exceeds max.poll.interval.ms
- Long garbage collection pauses that delay heartbeats
- Rolling deployments that restart consumers one after another
- Autoscaling rules that add and remove consumers too often
Reduce Rebalance Problems
Use Static Membership
Assign each consumer a fixed identity with group.instance.id. A restarted consumer that returns within the session timeout reclaims its old partitions without triggering a rebalance.
group.instance.id=billing-worker-1 session.timeout.ms=60000
Shorten Processing Per Poll
Lower max.poll.records so that each batch finishes well within the poll interval. Offload heavy work to a separate thread pool when needed.
Handle Rebalance Events in Code
A rebalance listener runs code just before partitions leave a consumer. Commit offsets and flush local state inside that callback so the next owner starts from the correct position.
Monitoring Tip
Run kafka-consumer-groups.sh with the describe and state options to see whether a group is stable or rebalancing. Alert on rebalance counts per hour to catch trouble early.