Kafka Brokers and Clusters
A broker is a single Kafka server. It accepts messages from producers, writes them to disk, and hands them to consumers on request. A cluster joins several brokers into one system that shares the work. Picture a city postal service: each post office branch handles a share of the mail, and the whole network keeps delivering when one branch closes for repairs.
What a Broker Does
Every broker performs four jobs at the same time.
- Receives write requests from producers
- Stores partition data in log files on local disk
- Serves read requests from consumers
- Copies partition data to other brokers for safety
Each broker carries a unique numeric ID. The ID stays the same after a restart, so the cluster always recognizes the server.
How a Cluster Shares Work
Kafka spreads the partitions of a topic across all brokers. The diagram below shows a topic named orders with three partitions on three brokers. Every partition has one leader copy and one follower copy.
+----------------+ +----------------+ +----------------+
| Broker 1 | | Broker 2 | | Broker 3 |
|----------------| |----------------| |----------------|
| orders-P0 (L) | | orders-P1 (L) | | orders-P2 (L) |
| orders-P2 (F) | | orders-P0 (F) | | orders-P1 (F) |
+----------------+ +----------------+ +----------------+
L = Leader copy F = Follower copy
Producers write to leaders only. Followers copy the data and wait. When Broker 2 stops, Kafka promotes the follower of orders-P1 on Broker 3 to leader, and traffic continues after a short pause.
Bootstrap Servers
A client does not need the address of every broker. It connects to one or two bootstrap servers and asks for the full cluster map. This map, called metadata, lists every topic, every partition, and the broker that leads each partition. The client then talks directly to the correct broker for each message.
List at least two bootstrap addresses in client settings. If the first broker is down at startup, the client tries the next address on the list.
The Controller Role
One member of the cluster takes the controller role. The controller watches the health of all brokers, elects new partition leaders after a failure, and announces changes to the rest of the cluster. When the controller fails, the cluster elects a replacement within seconds. In KRaft mode, a small group of controller nodes shares this job and keeps the cluster map consistent.
Important Broker Settings
| Setting | Purpose | Example |
|---|---|---|
| node.id | Unique number for the broker | 1 |
| listeners | Addresses the broker listens on | PLAINTEXT://:9092 |
| advertised.listeners | Address clients receive from metadata | PLAINTEXT://broker1:9092 |
| log.dirs | Folder that stores partition data | /var/kafka/data |
| num.partitions | Default partition count for new topics | 3 |
Cluster Size Guidelines
| Environment | Brokers | Reason |
|---|---|---|
| Personal laptop | 1 | Simplest way to learn |
| Testing | 3 | Practice replication and failure |
| Small production | 3 to 5 | Survives one broker failure |
| Large production | 6 or more | Spreads heavy traffic |
Common Mistakes
- Setting advertised.listeners to localhost, which breaks connections from other machines
- Giving two brokers the same node ID
- Storing data on a small shared disk that fills without warning
- Running a production cluster with one broker and no copies of the data
Quick Summary
A broker stores and serves data. A cluster combines brokers so that work spreads evenly and failures stay harmless. Clients learn the layout from bootstrap servers, and the controller keeps leadership current. A solid grasp of these roles makes every later setting easier to read.