Kafka In-Sync Replicas

Every partition keeps several copies called replicas. The in-sync replica set, shortened to ISR, lists the replicas that hold an up-to-date copy of the leader's data. Only members of this set may take over leadership without losing messages. Think of a group of students copying notes from a teacher: those who keep pace with the board stay in the "ready" group, and those who fall far behind step out until they catch up.

What Belongs in the ISR

The leader is always a member of the ISR. A follower stays in the set while it keeps fetching data and remains close to the leader. A follower that stops fetching or lags beyond the allowed time leaves the set. The controller records every change.

ISR in a Picture

  Partition orders-P0      Replication factor = 3

  Broker 1 (Leader)    offset: 1000   ---> in ISR
  Broker 2 (Follower)  offset:  998   ---> in ISR (close behind)
  Broker 3 (Follower)  offset:  400   ---> OUT of ISR (far behind)

  ISR = { Broker 1, Broker 2 }

Broker 3 might suffer from a slow disk or a network problem. Kafka removes it from the ISR so that it cannot block writes, and it rejoins automatically after catching up.

The Lag Rule

The setting replica.lag.time.max.ms defines how long a follower may go without catching up to the leader's log end. The default is 30 seconds. A follower that exceeds this limit leaves the ISR.

Commit Point and High Watermark

A message counts as committed once every replica in the ISR has stored it. The highest committed offset is called the high watermark. Consumers read only up to this point, so they never see data that could vanish after a leader change.

  Leader log   : [0][1][2][3][4][5][6]
  ISR synced to: [0][1][2][3][4]
                              ^
                        high watermark
  Consumers can read offsets 0 to 4 only.

Teaming Up with acks and min.insync.replicas

Two settings decide how safe a write is.

Setting	Where	Role
acks=all	Producer	Waits for every ISR member to store the message
min.insync.replicas	Topic or broker	Smallest ISR size that still accepts writes

With a replication factor of 3, acks=all, and min.insync.replicas=2, each acknowledged message lives on at least two brokers. One broker can fail without data loss and without stopping writes.

Scenario Table

ISR Size	Result with min.insync.replicas=2
3	Writes succeed, full safety
2	Writes succeed, one failure from the limit
1	Writes with acks=all fail with NotEnoughReplicas, reads continue

Rejecting writes in the last case protects data. The cluster prefers a short outage over silent risk.

Leader Election and ISR

When a leader fails, the controller picks a new leader from the ISR. Every ISR member holds all committed messages, so nothing committed disappears.

Unclean Leader Election

Suppose the ISR becomes empty because every in-sync replica died. The setting unclean.leader.election.enable decides what happens next. A value of false keeps the partition offline until an in-sync replica returns. A value of true allows an out-of-sync replica to lead, which restores service quickly and permanently loses the messages it missed. The default is false, and most production systems keep it that way.

Monitor the ISR

• Track the metric UnderReplicatedPartitions, which counts partitions with a shrunken ISR
• Track IsrShrinksPerSec and IsrExpandsPerSec to spot unstable followers
• Run kafka-topics.sh with the describe and under-replicated-partitions options for a quick check

A steady non-zero count of under-replicated partitions signals disk, network, or load trouble that deserves attention.

Practical Advice

• Use replication factor 3 and min.insync.replicas 2 for important topics
• Keep acks=all on producers that carry valuable data
• Investigate any broker that repeatedly leaves the ISR
• Avoid unclean leader election on topics where loss is unacceptable