Kafka Log Compaction
Standard retention deletes messages by age or size. Log compaction follows a different rule: it keeps the latest message for every key and removes older messages with the same key. The topic becomes a living snapshot of the newest state for each item, much like a contact list on a phone that stores only the current number for each person and drops the old ones.
Why Compaction Exists
Some data describes the current state of something rather than a stream of events. A customer profile, a product price, and a device setting all fit this pattern. Readers want the newest value, and old values waste space. Compaction trims the history while preserving the final answer for every key.
Before and After Compaction
The diagram shows a topic that stores user email addresses. The key is the user name.
BEFORE compaction (offset : key = value) 0 : anna = anna@old.com 1 : ravi = ravi@mail.com 2 : anna = anna@new.com 3 : mia = mia@mail.com 4 : ravi = ravi@work.com 5 : anna = anna@final.com AFTER compaction 3 : mia = mia@mail.com 4 : ravi = ravi@work.com 5 : anna = anna@final.com
Three messages survive, one per key. Offsets stay unchanged, so offset 3, 4, and 5 keep their original numbers. Gaps appear in the sequence, and consumers handle them without trouble.
Enable Compaction
Set the cleanup policy when creating the topic.
bin/kafka-topics.sh --create --topic user-emails \ --partitions 3 --replication-factor 3 \ --config cleanup.policy=compact \ --bootstrap-server localhost:9092
Every message in a compacted topic must carry a key. A message without a key cannot be matched to a newer version, so Kafka rejects it.
How the Cleaner Works
A background thread called the log cleaner scans each partition. It splits the log into two zones.
- The clean zone holds segments that were compacted before
- The dirty zone holds new segments not yet compacted
The cleaner builds a map of the latest offset for each key in the dirty zone and rewrites segments without the outdated records. The active segment, which still receives writes, stays untouched until it rolls over.
Deleting a Key with a Tombstone
A message with a key and a null value acts as a tombstone. It tells Kafka that the key no longer exists. Compaction first keeps the tombstone so that every consumer can see the deletion, then removes it after the delete.retention.ms period passes. The default period is 24 hours.
Key = anna Value = null --> tombstone After delete.retention.ms --> anna disappears from the log
Key Settings
| Setting | Meaning | Typical Value |
|---|---|---|
| cleanup.policy | Selects compact, delete, or both | compact |
| min.cleanable.dirty.ratio | Share of dirty data that triggers cleaning | 0.5 |
| segment.ms | Time before the active segment rolls | 604800000 |
| delete.retention.ms | How long tombstones stay visible | 86400000 |
| min.compaction.lag.ms | Minimum age before a message may be compacted | 0 |
Combine Compaction with Deletion
Set cleanup.policy to compact,delete to apply both rules. Kafka keeps the latest value per key and also removes any record older than the retention time. This mix suits data that should expire after a long period, such as session records.
Real Uses
- Rebuilding a cache after a restart by reading the compacted topic from the start
- Storing application settings where only the current setting matters
- Tracking the latest status of each order or device
- Backing the internal changelog topics of Kafka Streams
Kafka itself relies on compaction for the __consumer_offsets topic, which stores the latest committed position of every consumer group.
Limits to Remember
Compaction does not run instantly. Old values remain readable until the cleaner reaches them. Plan applications to tolerate duplicate keys during that window, and never use compaction as a guaranteed immediate delete.