DE Apache Kafka Basics
Apache Kafka is the backbone of real-time data infrastructure at thousands of organizations worldwide. Originally built at LinkedIn to handle billions of events per day, Kafka became an open-source project in 2011 and is now the standard platform for streaming data, event-driven pipelines, and real-time analytics. Data engineers who work with streaming data encounter Kafka almost universally.
What Kafka Does
Kafka is a distributed event streaming platform. It ingests high-volume streams of events from producers, stores them reliably and durably, and delivers them to consumers in real time or on demand. Unlike a traditional message queue that deletes messages after delivery, Kafka retains events for a configurable period — days, weeks, or indefinitely. Multiple consumers read the same event stream independently, each at their own pace.
The Highway System Analogy
Think of Kafka as a highway interchange. Vehicles (events) enter the interchange from multiple on-ramps (producers). The interchange routes them to multiple exits (consumers) simultaneously. Each exit leads to a different destination — a data warehouse, a fraud detection system, a real-time dashboard. Vehicles travel at different speeds on different exits. The highway itself does not disappear once vehicles pass — it stays open, and vehicles can revisit the same route. Kafka stores events so consumers can "replay" past events when needed.
Core Kafka Components
Topics
A topic is a named, ordered log of events. Producers write events to a specific topic. Consumers subscribe to topics they care about. A topic is comparable to a database table, but instead of storing current state, it stores a chronological sequence of events.
Topics in an e-commerce system: orders.created -- new order events orders.updated -- order status change events payments.completed -- successful payment events inventory.updated -- stock level change events users.registered -- new account events
Partitions
Each topic divides into one or more partitions. A partition is an ordered, immutable sequence of records. Events within one partition maintain strict order. Partitioning enables parallelism — multiple consumer instances process different partitions simultaneously. More partitions allow more parallel consumers and higher throughput.
Topic: orders.created with 3 partitions: Partition 0: Event1(ORD100) --> Event4(ORD103) --> Event7(ORD106) Partition 1: Event2(ORD101) --> Event5(ORD104) --> Event8(ORD107) Partition 2: Event3(ORD102) --> Event6(ORD105) --> Event9(ORD108) 3 consumer instances can each read one partition in parallel. Throughput = 3x a single-partition topic.
Brokers
Kafka brokers are the server processes that store and serve event data. A Kafka cluster consists of multiple brokers. Each broker stores some partitions of each topic. Distributing partitions across brokers provides fault tolerance — if one broker fails, the partitions it hosted continue serving from replicas on other brokers.
Producers
Producers write events to Kafka topics. They choose which partition to write to — randomly (round-robin), by a user-defined key (same key always goes to same partition), or by custom logic. Writing events with a key ensures that all events for the same entity (e.g., the same customer_id) land in the same partition and stay in order.
Consumers
Consumers read events from topics. Each consumer tracks its position in the partition using an offset — a sequential number assigned to each event. Consumers periodically commit their offset to Kafka. If a consumer crashes and restarts, it resumes from the last committed offset rather than starting from the beginning or losing events.
Partition with offsets:
Offset: 0 1 2 3 4
Event: [ORD100] [ORD101] [ORD102] [ORD103] [ORD104]
^
Consumer committed offset=2
On restart, reads from offset=3
Consumer Groups
A consumer group is a set of consumers that collectively read a topic. Kafka assigns each partition to exactly one consumer within the group. This enables load balancing — a topic with 6 partitions and a consumer group with 3 consumers assigns 2 partitions to each consumer. Adding more consumers to the group (up to the number of partitions) increases throughput linearly.
Consumer Group with 3 consumers on a 6-partition topic: Consumer A --> Partitions 0, 1 Consumer B --> Partitions 2, 3 Consumer C --> Partitions 4, 5 All 3 consumers read simultaneously -- 3x throughput vs. single consumer.
Kafka's Retention Model
Kafka retains events for a configurable period regardless of whether consumers have read them. Setting a 7-day retention means every event stays available for replay for one week. A new consumer joining the group can read all events from the beginning. This makes Kafka different from traditional message queues that delete messages after delivery — Kafka acts as a distributed, persistent event log.
Replication for Fault Tolerance
Each partition replicates to multiple brokers. The replication factor defines how many copies exist. A replication factor of 3 means three brokers each hold a copy of every partition. If two brokers fail simultaneously, data remains available from the third. One broker acts as the leader for each partition and handles all reads and writes; replicas stay in sync and take over if the leader fails.
Common Kafka Use Cases in Data Engineering
Use Case | How Kafka Helps
--------------------------|---------------------------------------------
Real-time data pipelines | Source systems publish events; warehouse
| consumers load in near real-time
Change Data Capture | Debezium publishes DB changes to Kafka topics
Log aggregation | Application logs stream to Kafka for analysis
Stream processing input | Flink/Spark Streaming read from Kafka topics
Microservice messaging | Services communicate through Kafka events
Activity tracking | User clicks/actions stream to analytics system
Summary
Apache Kafka is a distributed event streaming platform that stores event streams in partitioned, replicated topics. Producers write events; consumers read them at their own pace using offsets. Consumer groups enable parallel processing by distributing partitions across multiple consumer instances. Kafka's durable retention model allows event replay and late-joining consumers. Its high throughput, fault tolerance, and decoupled producer-consumer model make it the standard infrastructure for real-time data pipelines, event-driven architectures, and streaming analytics.