Kafka Offsets Explained Reading Messages in Order
The offset is one of Kafka's most powerful and distinctive concepts. It is the mechanism that lets consumers read messages in order, track progress through a topic, recover from failures without data loss, and replay historical events. Once you truly understand offsets, you understand how Kafka handles state, ordering, and fault recovery — all at once.
What Is an Offset
An offset is a unique, sequential integer assigned to each message within a partition. The first message in a partition gets offset 0. The second gets offset 1. The hundredth gets offset 99. Offsets are immutable — once assigned, they never change, even if messages are later deleted due to retention.
Offsets are partition-specific. The same offset number (say, offset 42) exists independently in every partition of a topic. Offset 42 in partition 0 is a completely different message from offset 42 in partition 1. When referring to a message's position in Kafka, you always specify both the partition and the offset.
TOPIC: orders (3 partitions) Partition 0: [msg@0] [msg@1] [msg@2] [msg@3] [msg@4] ... Partition 1: [msg@0] [msg@1] [msg@2] ... Partition 2: [msg@0] [msg@1] [msg@2] [msg@3] ... To uniquely identify any message: Topic: orders + Partition: 1 + Offset: 2 → specific unique message Topic: orders + Partition: 0 + Offset: 2 → different unique message
Who Assigns Offsets
The Kafka broker assigns offsets, not the producer. When a producer sends a message to a partition, the broker appends it to the partition log and assigns the next sequential offset. The broker sends the assigned offset back to the producer as confirmation. The producer cannot choose what offset a message gets — that is entirely the broker's responsibility.
This broker-controlled assignment guarantees that offsets within a partition are always sequential, without gaps (barring compaction on compacted topics), and that no two messages in the same partition ever share an offset.
Consumer Offset: Tracking Your Reading Position
While the partition offset identifies a message's position in the log, the consumer offset tracks where a particular consumer (or consumer group) is in its reading of that partition.
Think of reading a book. The page number (like a partition offset) is printed in the book and never changes. Your bookmark is the consumer offset — it marks where you stopped reading. You control where your bookmark is. Move it forward as you read. Move it backward if you want to reread a chapter.
CONSUMER READING PROGRESS METAPHOR:
Partition 0 messages:
[msg@0][msg@1][msg@2][msg@3][msg@4][msg@5][msg@6][msg@7]
↑
Consumer A's current position
Consumer A has read up to offset 3
Consumer A's committed offset = 4 (next to read)
Consumer B (different group) reads independently:
[msg@0][msg@1][msg@2][msg@3][msg@4][msg@5][msg@6][msg@7]
↑
Consumer B at offset 7
Consumer B is nearly caught up
Current Offset vs Committed Offset
Two offset values matter for each consumer-partition combination:
Current offset: The offset of the next message the consumer will fetch from the broker. It advances in memory as the consumer reads. The broker doesn't know about the current offset until the consumer commits.
Committed offset: The offset the consumer has explicitly committed to Kafka's internal topic (__consumer_offsets). It represents the last successfully processed message. If the consumer crashes and restarts, it resumes from the committed offset — not the current offset. Any messages between the committed offset and the current offset at time of crash will be reprocessed.
COMMITTED vs CURRENT OFFSET: Partition 0: [msg@0][msg@1][msg@2][msg@3][msg@4][msg@5] Consumer reads msg@0, msg@1, msg@2, msg@3 in memory (current offset = 4) Consumer commits offset 2 (committed offset = 3, meaning "read up to 2") CONSUMER CRASHES at this point. On restart: Consumer resumes from committed offset 3 (the next to read after 2) Consumer re-reads msg@3 and msg@4 (they were already read but not committed) This causes duplicate processing of msg@3 (at-least-once delivery) Lesson: Commit frequently to minimize reprocessing on failure.
Where Offsets Are Stored
Consumer group offsets are stored in Kafka's internal topic called __consumer_offsets. This topic has 50 partitions by default and uses the same replication mechanism as any Kafka topic, making offset storage durable and fault-tolerant.
When a consumer commits an offset, it produces a record to __consumer_offsets containing: the consumer group name, the topic name, the partition number, and the committed offset value. On consumer restart, Kafka looks up the latest committed offset for that consumer group and partition and tells the consumer where to start reading.
__consumer_offsets topic stores entries like: KEY: [group="analytics-team", topic="orders", partition=0] VALUE: [offset=142, metadata="optional string", timestamp=1709654400] KEY: [group="analytics-team", topic="orders", partition=1] VALUE: [offset=89, metadata="", timestamp=1709654398] KEY: [group="email-service", topic="orders", partition=0] VALUE: [offset=141, metadata="", timestamp=1709654399]
Reading From a Specific Offset
Consumers don't always have to start from where they left off. Kafka allows consumers to seek to any offset in any partition — past, present, or a special position.
Special Offset Positions
earliest (beginning): Start reading from offset 0 — the very first message still available in the partition. If the oldest messages have expired due to retention, "earliest" means the first available message, not necessarily offset 0.
latest (end): Start reading from the latest offset — only process messages that arrive after the consumer starts. No historical data is processed.
Specific offset: Seek to any specific numeric offset and read from there. Useful for replaying events from a known point in time.
Timestamp-based: Ask Kafka "what is the first offset at or after this timestamp?" Kafka uses the time index to find the offset and seeks to it. Useful for replaying "all events from the last 24 hours."
OFFSET STARTING POSITIONS: Partition 0 (contains 1000 messages, oldest 200 expired): [expired][expired][msg@200][msg@201]...[msg@998][msg@999][new→] auto.offset.reset=earliest: Start at msg@200 (first available) auto.offset.reset=latest: Start at new→ (only future messages) seekToOffset(500): Start at msg@500 specifically seekToTimestamp(T): Find first offset where timestamp ≥ T, start there
The auto.offset.reset Property
When a consumer group reads a topic for the first time (no committed offset exists yet) or when the committed offset is no longer available (message expired), Kafka needs to know where to start. The auto.offset.reset property on the consumer controls this.
earliest: Start from the beginning. Used when you want to process all historical events.
latest: Start from now. Used when you only care about new events.
none: Throw an exception if no committed offset is found. Used in strict environments where missing data is a critical error.
How Offset Commit Works in Practice
Kafka consumer clients handle offset commitment automatically by default, but you can control the process manually for more precise delivery guarantees.
Auto-Commit Mode
When enable.auto.commit=true (the default), the consumer automatically commits the current offset every auto.commit.interval.ms milliseconds (default 5 seconds). This is convenient but creates a window for duplicate processing — if the consumer crashes between commits, the messages since the last commit are reprocessed on restart.
AUTO-COMMIT TIMELINE:
t=0s: Consumer reads msg@100, 101, 102, 103...
t=5s: AUTO-COMMIT: committed offset = 108 (wherever consumer is at t=5s)
t=8s: Consumer reads msg@108, 109, 110...
t=10s: AUTO-COMMIT: committed offset = 111
t=11s: CONSUMER CRASHES
t=15s: Consumer restarts → resumes from offset 111 (no messages lost)
If crash happened at t=9s instead:
t=9s: CRASH. Consumer read 108, 109 but no commit happened yet
t=15s: Consumer restarts → resumes from offset 108 (reprocesses 108, 109)
This is at-least-once delivery (duplicates possible)
Manual Commit Mode
For stronger delivery guarantees, disable auto-commit and commit manually after successfully processing each batch. This gives precise control over what "processed" means.
Two manual commit methods exist:
commitSync(): Blocks until the commit succeeds. If the commit fails, it retries. Safer but slower — the consumer waits for the commit acknowledgment before processing more messages.
commitAsync(): Non-blocking. Sends the commit and continues processing. If the commit fails, it does not automatically retry (to avoid committing a stale offset after a newer commit succeeds). Faster but requires careful failure handling.
MANUAL COMMIT PATTERN (Java pseudocode): consumer.poll() returns records [msg@50, msg@51, msg@52] Process msg@50 → success Process msg@51 → success Process msg@52 → success consumer.commitSync() ← blocks, commits offset=53 (next to read) Next poll returns [msg@53, msg@54, msg@55] ... If crash between processing and commit: Restart → resumes from 50 → reprocesses 50, 51, 52 (at-least-once) If crash during processing of msg@51: Restart → resumes from 50 → reprocesses from 50 (at-least-once)
Offset Lag: A Critical Metric
Consumer lag (or offset lag) is the difference between the latest offset in a partition and the consumer group's committed offset for that partition. It tells you how far behind a consumer group is from the newest messages.
CONSUMER LAG CALCULATION: Partition 0 latest offset: 10,542 Consumer group committed offset: 9,800 Consumer lag: 742 messages behind Partition 1 latest offset: 8,320 Consumer group committed offset: 8,310 Consumer lag: 10 messages behind (nearly caught up) Total lag across all partitions: 752 messages Monitoring lag tells you: - Is your consumer keeping up with incoming data? - Is lag growing (consumer too slow) or shrinking (consumer recovering)? - Alert when lag exceeds a threshold → consumer needs attention
Monitoring Consumer Lag
Use the kafka-consumer-groups command to inspect consumer group offsets and lag:
bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --describe \ --group my-consumer-group Output: GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG my-consumer-group orders 0 142 142 0 my-consumer-group orders 1 89 101 12 my-consumer-group orders 2 201 201 0
Partition 1 has a lag of 12 — the consumer is 12 messages behind. This might be temporary (burst of messages) or indicate a problem (consumer is too slow). Monitoring this lag continuously in production is essential for healthy Kafka operations.
Offset Retention
Just as Kafka retains messages for a configurable period, it also retains consumer group offsets. The default retention for committed offsets is 7 days (offsets.retention.minutes=10080 on the broker). If a consumer group doesn't commit any offsets for 7 days, its stored offsets are deleted.
This matters if you have consumer groups that run infrequently — say, a batch job that runs weekly. When it returns after more than 7 days of inactivity, its committed offset is gone. Kafka falls back to the auto.offset.reset policy to determine where to start reading. Increase offsets.retention.minutes for consumer groups with long gaps between runs.
Replaying Events: The Power of Offsets
One of Kafka's most compelling features is the ability to replay any past event by resetting a consumer group's offset to an earlier position. This is only possible because offsets are durable and messages stay in Kafka until the retention period expires.
Practical replay use cases:
Bug fix replay: A consumer had a bug that corrupted processed data. Fix the bug, reset the consumer group offset to before the bug was introduced, replay all affected messages.
New consumer bootstrap: A new downstream system needs access to all historical events. Reset its consumer offset to 0 (or to the earliest available message) and let it catch up.
Analytics recomputation: Business logic changed and you need to recompute analytics from scratch. Reset the analytics consumer group offset to the beginning and let it reprocess everything.
RESETTING A CONSUMER GROUP OFFSET: # Reset to earliest (beginning of topic): bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group analytics-team \ --topic orders \ --reset-offsets \ --to-earliest \ --execute # Reset to specific offset: bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group analytics-team \ --topic orders \ --reset-offsets \ --to-offset 5000 \ --execute # Reset to timestamp (replay last 24 hours): bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 \ --group analytics-team \ --topic orders \ --reset-offsets \ --to-datetime 2024-03-05T00:00:00.000 \ --execute
Key Points
- An offset is a unique sequential integer assigned by the broker to each message within a partition. Offsets are immutable and partition-specific.
- Consumer offset tracks where a consumer group is in reading a partition. It is stored in Kafka's internal
__consumer_offsetstopic. - Auto-commit saves the offset automatically every N milliseconds. Manual commit gives precise control over what counts as "processed."
- Consumer lag is the difference between the latest partition offset and the consumer's committed offset. High and growing lag signals a problem.
- Consumers can seek to any offset — earliest, latest, a specific number, or a timestamp — enabling event replay without external systems.
- Offset retention defaults to 7 days. Consumer groups inactive for longer than that lose their saved position.