Apache Kafka Core Concepts
Three concepts sit at the structural heart of every Apache Kafka deployment: brokers, clusters, and the coordination layer (ZooKeeper or KRaft). These are the physical and logical components that make Kafka work at scale. Understanding these building blocks lets you reason about how Kafka handles failures, distributes load, and ensures no data is lost even when servers crash.
This topic treats each component as a role in an organization — with a clear job, a hierarchy, and a relationship to every other role.
What Is a Kafka Broker
A Kafka broker is a single Kafka server — one machine (physical or virtual) running the Kafka process. Every broker has a unique numeric identifier called a broker ID. In a cluster of three brokers, they might have IDs 1, 2, and 3.
The broker's job is to receive messages from producers, store them on disk, and serve them to consumers. It also maintains metadata about what topics exist, which partitions belong to it, and which partitions it is currently leading.
What a Broker Stores
Each broker stores partitions — segments of topic data — on disk. A topic with three partitions has each partition stored on a different broker (when the cluster has at least three brokers). The broker writes incoming messages to partition files called log segments. These are just files on the broker's file system.
BROKER 1 (ID: 1) BROKER 2 (ID: 2)
──────────────────── ────────────────────
/data/ /data/
orders-0/ orders-1/
00000000.log 00000000.log
00000000.index 00000000.index
payments-0/ payments-1/
00000000.log 00000000.log
Each folder = one partition.
Each .log file = the actual messages stored on disk.
Each .index file = maps offsets to byte positions for fast lookup.
How a Broker Handles a Producer Request
When a producer sends a message to a topic, the broker that is the leader for the target partition receives the message, appends it to the partition's log file, optionally waits for replica brokers to copy the message (based on acknowledgement settings), and sends a success response back to the producer.
PRODUCER → [Message: "order_123"] → BROKER 2 (leader of orders partition 1)
↓
Writes to orders-1/00000000.log
↓
Replica on BROKER 3 copies the message
↓
Sends ACK back to PRODUCER: "Stored!"
What Is a Kafka Cluster
A Kafka cluster is a group of brokers working together as a single system. From the outside, you interact with the cluster — you produce to the cluster, consume from the cluster — without caring which specific broker handles each request.
Why Clusters Exist
A single broker has limits: it has finite disk space, finite RAM, finite network bandwidth, and it can fail. A cluster solves all four problems:
Capacity: Data is distributed across all brokers. The cluster's total capacity is the sum of all broker capacities.
Throughput: Many brokers handle requests in parallel. A 10-broker cluster can handle roughly 10 times the throughput of a single broker.
Fault tolerance: If one broker crashes, the other brokers still have copies of the data (through replication) and keep serving requests. The cluster recovers automatically.
Availability: No single point of failure. The cluster stays up even when individual brokers go down for maintenance or failure.
Cluster Diagram
KAFKA CLUSTER (5 brokers) ──────────────────────────────────────────────────────────────────── BROKER 1 BROKER 2 BROKER 3 BROKER 4 BROKER 5 ──────── ──────── ──────── ──────── ──────── orders-P0 orders-P1 orders-P2 pays-P0 pays-P1 (leader) (leader) (leader) (leader) (leader) orders-P1 orders-P2 orders-P0 pays-P1 pays-P0 (replica) (replica) (replica) (replica) (replica) orders-P2 orders-P0 orders-P1 pays-P0 pays-P1 (replica) (replica) (replica) (replica) (replica) ──────────────────────────────────────────────────────────────────── Each partition has one leader and multiple replicas on different brokers. If Broker 2 goes down, Broker 1 or 3 takes over as leader for orders-P1.
The Bootstrap Server: Your Entry Point to the Cluster
When a producer or consumer wants to connect to Kafka, it needs at least one broker address to start with. This starting address is called the bootstrap server. The client connects to the bootstrap server, asks for cluster metadata (the list of all brokers and which partitions they hold), and then communicates directly with the appropriate brokers for its actual work.
You don't need to list every broker as a bootstrap server. One is enough to discover the rest. But listing two or three bootstrap servers makes your application more resilient — if one is temporarily down, the client falls back to another.
CLIENT STARTUP PROCESS: Step 1: Client connects to bootstrap-server: broker1:9092 Step 2: Client asks: "Who's in the cluster? What topics exist? Who leads which partition?" Step 3: Broker1 responds with cluster metadata (all brokers, all topics, all partition leaders) Step 4: Client now talks directly to the right broker for each partition bootstrap-servers=broker1:9092,broker2:9092 ← list 2-3 for resilience
ZooKeeper: The Traditional Cluster Coordinator
In Kafka versions before 2.8, every Kafka cluster required Apache ZooKeeper to run alongside it. ZooKeeper's job was to manage cluster metadata and coordinate the brokers — a job that brokers cannot reliably do for themselves without an external coordinator.
What ZooKeeper Did for Kafka
ZooKeeper stored the source of truth about the Kafka cluster. Every time the cluster state changed — a broker joined, a broker crashed, a partition leadership changed — ZooKeeper recorded the new state and notified affected brokers.
The specific tasks ZooKeeper handled include:
Broker Registry: When a broker starts, it registers itself with ZooKeeper. When a broker crashes, ZooKeeper detects the lost connection and notifies the cluster controller.
Leader Election: For each partition, one broker is the leader. ZooKeeper managed the election of new leaders when the current leader failed.
Topic Configuration: ZooKeeper stored the list of topics, their partition counts, replication factors, and configuration overrides.
Consumer Group Coordination: Older Kafka versions stored consumer group offsets in ZooKeeper. Newer versions moved this to a Kafka topic, but ZooKeeper still played a coordination role.
ZOOKEEPER'S ROLE IN KAFKA CLUSTER (Traditional Setup) ZooKeeper Cluster (3 nodes for ZK HA): ZK Node 1 ZK Node 2 ZK Node 3 ───────── ───────── ───────── ZooKeeper knows: /brokers/ids/1 → "broker1.example.com:9092" /brokers/ids/2 → "broker2.example.com:9092" /topics/orders/partitions/0/state → "leader: 1" /topics/orders/partitions/1/state → "leader: 2" ↕ All Kafka brokers register with and watch ZooKeeper KAFKA CLUSTER (3 brokers): BROKER 1 BROKER 2 BROKER 3 ───────── ───────── ─────────
ZooKeeper's Architecture
ZooKeeper itself runs as a cluster (called an ensemble) for high availability. ZooKeeper ensembles must have an odd number of nodes — 3, 5, or 7 — because they use a majority voting system. A 3-node ZooKeeper ensemble tolerates 1 node failure. A 5-node ensemble tolerates 2 failures.
Running ZooKeeper alongside Kafka meant operating two separate distributed systems — double the operational complexity. This is one of the major reasons Kafka is transitioning away from ZooKeeper.
KRaft Mode: Kafka Without ZooKeeper
Starting with Kafka 2.8 (preview) and becoming the default in Kafka 3.3+, KRaft mode (Kafka Raft Metadata mode) removes the need for ZooKeeper entirely. Kafka now manages its own cluster metadata using the Raft consensus protocol built directly into the broker.
How KRaft Works
In KRaft mode, a subset of brokers take on the role of "controllers." These controllers collectively manage cluster metadata using the Raft protocol — a distributed algorithm that elects a leader, replicates decisions to followers, and handles leader failures automatically.
A broker can be a controller, a regular data broker, or both (in small clusters). In large clusters, dedicated controller nodes handle only metadata — no data serving — for maximum performance and reliability.
KRAFT MODE CLUSTER ARCHITECTURE
CONTROLLER NODE 1 CONTROLLER NODE 2 CONTROLLER NODE 3
(active controller) (follower) (follower)
───────────────── ───────────────── ─────────────────
Manages metadata Stays in sync Stays in sync
Runs Raft leader Ready to take over Ready to take over
↕ metadata updates
BROKER 4 BROKER 5 BROKER 6 BROKER 7
───────── ───────── ───────── ─────────
Stores data Stores data Stores data Stores data
Controllers hold all cluster metadata.
If Controller 1 fails, Raft elects Controller 2 instantly.
No ZooKeeper. No external coordination system needed.
Benefits of KRaft Over ZooKeeper
Simpler operations: One system to deploy, monitor, and maintain instead of two (Kafka + ZooKeeper).
Faster recovery: ZooKeeper-based clusters sometimes took tens of seconds to elect a new partition leader after a broker failure. KRaft reduces this to milliseconds in most cases.
Larger clusters: ZooKeeper struggled with very large Kafka clusters (hundreds of thousands of partitions). KRaft scales to millions of partitions by redesigning how metadata is stored and replicated.
Consistent architecture: Using the Raft protocol inside Kafka itself makes the entire system's behavior more predictable and easier to reason about.
The Kafka Controller Broker
In both ZooKeeper mode and KRaft mode, one broker (or one controller node in KRaft) acts as the cluster controller at any given time. The controller has elevated responsibilities beyond regular brokers.
What the Controller Does
When a broker crashes, the controller detects the failure (through ZooKeeper in traditional mode, or through Raft heartbeats in KRaft mode) and reassigns partition leadership. The controller picks a new leader for each partition that the failed broker was leading, updates the cluster metadata, and notifies all remaining brokers about the change.
This process is called leader election or partition reassignment. It typically takes seconds and is completely transparent to producers and consumers — they receive temporary errors, retry automatically, and reconnect to the new leader without manual intervention.
BROKER FAILURE SCENARIO Before failure: Broker 2: leader of orders-P1 ← CRASHES Controller detects failure via: [ZooKeeper mode] ZooKeeper signals: "Broker 2 disconnected" [KRaft mode] Raft leader detects: "Broker 2 heartbeat lost" Controller elects new leader: Controller picks Broker 3 (was replica for orders-P1) Broker 3 becomes new leader of orders-P1 Controller notifies all brokers of new leader. Producers and consumers retry → now connect to Broker 3. Total downtime for orders-P1: a few seconds maximum.
Broker Configuration Essentials
The Kafka broker's behavior is controlled through the server.properties file (or kraft/server.properties for KRaft mode). Understanding the key configuration properties helps you understand how the broker works.
broker.id: The unique integer identifier for this broker. Each broker in the cluster must have a different ID.
log.dirs: The directory (or directories) where Kafka stores its partition data on disk. For high performance, this is often pointed to fast SSD storage.
num.partitions: The default number of partitions for automatically created topics. Topics created manually override this default.
log.retention.hours: How long Kafka keeps messages on disk before deleting them. Default is 168 hours (7 days).
zookeeper.connect: (ZooKeeper mode only) The address of the ZooKeeper ensemble, e.g., zk1:2181,zk2:2181,zk3:2181.
listeners: The network address and port the broker listens on. Default is PLAINTEXT://:9092.
advertised.listeners: The address that the broker announces to clients. Critical for cloud deployments where the internal address differs from the external address.
How Many Brokers Do You Need
The answer depends on your requirements for replication, throughput, and availability.
A minimum production cluster has 3 brokers. With 3 brokers, you can use a replication factor of 3 (the standard recommendation), tolerate 1 broker failure, and maintain partition leadership election with the remaining 2 brokers.
For high throughput, add brokers proportional to your throughput requirements. Kafka scales linearly — 6 brokers provide roughly twice the throughput of 3 brokers for the same workload.
For availability during rolling upgrades or maintenance, having more brokers means you can take one down without reducing cluster capacity significantly.
BROKER COUNT DECISION GUIDE 1 broker: Development/testing only. No fault tolerance. 3 brokers: Minimum for production. RF=3, tolerate 1 failure. 5 brokers: Better for large workloads. RF=3, tolerate 2 failures. 10+ brokers: High-throughput production. Scale partitions across brokers. Rule of thumb: Replication factor should be 3. You need at least as many brokers as your highest replication factor.
Key Points
- A Kafka broker is a single Kafka server identified by a unique broker ID. It stores partition data on disk and serves producer and consumer requests.
- A Kafka cluster is a group of brokers working together. Clusters provide capacity, throughput, fault tolerance, and high availability.
- Clients use a bootstrap server to discover the full cluster topology before connecting to individual brokers.
- ZooKeeper (traditional) manages cluster metadata, broker registration, and leader election for Kafka from the outside.
- KRaft mode (modern) builds cluster coordination directly into Kafka using the Raft consensus protocol, eliminating the need for ZooKeeper.
- One broker (or controller node in KRaft) acts as the cluster controller, handling partition leader elections when brokers fail.
- Minimum production setup: 3 brokers with replication factor 3 to tolerate 1 broker failure.