DE Kafka Producers and Consumers
Understanding Kafka at a conceptual level is a starting point. Building production systems requires knowing how producers and consumers actually work — how producers guarantee delivery, how consumers track progress, how both handle failures, and how to tune them for throughput or latency. These details separate a working proof-of-concept from a reliable production pipeline.
Kafka Producers in Depth
A Kafka producer is any application that writes events to a Kafka topic. The producer connects to the Kafka cluster, serializes the event data into bytes, selects the target partition, and sends the event. Kafka's producer API handles batching, compression, retries, and acknowledgment internally.
Partitioning Logic
When a producer sends a message with a key, Kafka hashes the key and routes the message to a consistent partition. All messages with the same key always land in the same partition — maintaining order for that key across all messages. When no key is specified, Kafka distributes messages across partitions in round-robin fashion.
Partitioning by Key:
Producer sends orders with customer_id as key:
customer_id="C001" --> hash("C001") % 3 = 1 --> Partition 1
customer_id="C001" --> hash("C001") % 3 = 1 --> Partition 1
customer_id="C002" --> hash("C002") % 3 = 0 --> Partition 0
customer_id="C003" --> hash("C003") % 3 = 2 --> Partition 2
Result: All orders for C001 land in Partition 1, always in order.
A consumer reading Partition 1 sees C001 events in arrival sequence.
Acknowledgment (acks) Settings
The acks setting controls how many broker acknowledgments the producer waits for before considering a message successfully sent. This setting directly trades durability against throughput.
acks=0 : Producer sends and forgets. No acknowledgment waited.
Highest throughput. Risk: messages can be lost if broker fails.
Use for: Non-critical logs, metrics where some loss is acceptable.
acks=1 : Producer waits for acknowledgment from the leader broker only.
Moderate throughput. Risk: data loss if leader fails before replication.
Use for: General purpose; balanced trade-off.
acks=all: Producer waits for acknowledgment from all in-sync replicas.
Lowest throughput. No data loss as long as one replica survives.
Use for: Financial data, order events, critical pipelines.
Batching and Compression
Kafka producers batch multiple messages together before sending. Sending 1,000 messages as one batch is far more efficient than 1,000 individual network calls. The linger.ms setting controls how long the producer waits to fill a batch before sending. Higher values increase latency slightly but improve throughput significantly for high-volume systems.
Producer Tuning for Throughput: batch.size=1048576 # 1MB batch size linger.ms=20 # Wait 20ms to fill batch compression.type=snappy # Compress batches (reduces network I/O) Producer Tuning for Low Latency: batch.size=16384 # Small batch size linger.ms=0 # Send immediately (no waiting) compression.type=none # Skip compression overhead
Idempotent Producer
Without idempotency, network retries can cause duplicate messages. If the broker received a message but its acknowledgment was lost in transit, the producer retries and the broker stores it twice. Enabling the idempotent producer gives each message a unique sequence number; the broker deduplicates automatically.
Enable idempotent producer: enable.idempotence=true Effect: Each message gets a producer ID + sequence number. Broker rejects duplicate sequence numbers from the same producer. Result: Exactly-once delivery from producer to broker.
Kafka Consumers in Depth
A Kafka consumer reads events from one or more partitions. It tracks its position using offsets and commits those offsets to Kafka so it can resume after a restart without reprocessing or losing events.
Poll Loop
Consumer applications run a continuous poll loop. The consumer calls poll() to fetch a batch of records from Kafka, processes them, commits the offset, and calls poll() again. The poll interval must stay frequent enough to prevent Kafka from considering the consumer dead and reassigning its partitions.
Consumer Poll Loop (Python using confluent-kafka):
from confluent_kafka import Consumer
consumer = Consumer({
'bootstrap.servers': 'kafka-broker:9092',
'group.id': 'warehouse-loader-group',
'auto.offset.reset': 'earliest'
})
consumer.subscribe(['orders.created'])
while True:
msg = consumer.poll(timeout=1.0) # Wait up to 1s for a message
if msg is None:
continue
if msg.error():
handle_error(msg.error())
continue
event = json.loads(msg.value()) # Deserialize the event
load_to_warehouse(event) # Process the event
consumer.commit() # Commit offset after success
Offset Management
Offsets track how far a consumer group has read in each partition. Committed offsets persist in Kafka's internal __consumer_offsets topic. On restart, the consumer fetches the last committed offset and resumes from there.
Offset Commit Strategies:
Auto-commit (enable.auto.commit=true):
Kafka commits offsets automatically every 5 seconds.
Risk: Consumer crashes after commit but before processing --
those messages are skipped (at-most-once delivery).
Manual commit after processing:
Consumer commits only after successfully processing each batch.
Guarantees at-least-once delivery (may reprocess on crash).
Code: consumer.commit() after processing each poll() result.
Transactional (exactly-once):
Commit offset and write output atomically in one transaction.
Guarantees each event processed exactly once.
Requires Kafka transactions + idempotent consumer logic.
Consumer Group Rebalancing
When a consumer joins or leaves a group, Kafka reassigns partitions among the remaining members. During a rebalance, all consumers in the group pause, partitions redistribute, and processing resumes. Rebalances cause brief processing gaps. Data engineers minimize rebalances by keeping consumer instances stable and tuning heartbeat and session timeout settings.
Consumer Group Rebalance Scenario: Before: 3 consumers, 6 partitions Consumer A: Partitions 0, 1 Consumer B: Partitions 2, 3 Consumer C: Partitions 4, 5 Consumer B crashes --> Rebalance triggers After: 2 consumers, 6 partitions Consumer A: Partitions 0, 1, 2 Consumer C: Partitions 3, 4, 5
Delivery Semantics
Semantic | How Achieved | Risk
------------------|----------------------------|---------------------------
At-most-once | Auto-commit before process | Data loss on crash
At-least-once | Manual commit after process| Duplicate processing on crash
Exactly-once | Idempotent producer + | No loss, no duplicates
| transactional API | Higher complexity and cost
Dead Letter Queues
Some events cannot be processed — malformed JSON, unexpected schema, business logic violations. Rather than failing the entire consumer or silently dropping bad events, producers send failed events to a separate "dead letter" topic. An engineer inspects and replays or discards events from the dead letter queue after investigation.
Dead Letter Queue Pattern:
Normal flow: orders.created --> Consumer --> Data Warehouse
Failure flow: orders.created --> Consumer --> FAILS to parse
--> Publish to orders.created.dlq
--> Engineer investigates dlq
--> Fix and replay valid events
Monitoring Producers and Consumers
Consumer lag is the most critical metric to monitor — it measures how far behind consumers are from the latest events in the topic. Growing lag means the consumer cannot keep up with the production rate and real-time guarantees are breaking down.
Consumer Lag Monitoring: Topic: orders.created, 6 partitions Latest offset per partition: [1500, 1480, 1520, 1490, 1510, 1500] Consumer committed offset: [1498, 1478, 1520, 1487, 1508, 1499] Lag per partition: [ 2, 2, 0, 3, 2, 1] Total lag: 10 messages behind Alert: If total lag > 10,000 and growing -- consumer is falling behind
Summary
Kafka producers control delivery reliability through acknowledgment settings, batching, compression, and idempotency. Consumers maintain progress through offset management, choosing between at-most-once, at-least-once, or exactly-once delivery semantics. Consumer groups enable parallel processing by distributing partitions across multiple instances. Dead letter queues handle unprocessable events safely. Monitoring consumer lag is the primary signal of streaming pipeline health. Mastering producer and consumer configuration turns Kafka from a simple message bus into a reliable, high-throughput data backbone.