Installing and Setting Up Apache Kafka

Setting up Kafka on your local machine is the fastest way to go from theory to practice. Once Kafka is running locally, you can experiment freely — create topics, produce messages, consume them, and see exactly how each concept you have learned works in reality. This topic walks you through the complete setup process step by step.

Local Kafka is not meant for production. It is your personal laboratory. Use it to build confidence, run experiments, and test ideas before working with production clusters.

What You Need Before You Start

Kafka runs on Java. Before installing Kafka, make sure Java is installed on your machine. Kafka requires Java 8 or higher, but Java 11 or 17 LTS is recommended for stability and long-term support.

Checking Your Java Version

Open a terminal (Command Prompt on Windows, Terminal on Mac or Linux) and type:

java -version

You should see output similar to:

openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7)

If Java is not installed or the version is below 8, download and install the latest Java Development Kit (JDK) from adoptium.net or oracle.com/java. The OpenJDK distribution from Adoptium is free, open-source, and recommended.

System Requirements

For a local development setup, any modern computer works fine. You need at least 4 GB of RAM available (Kafka and ZooKeeper together use roughly 1–2 GB), a few gigabytes of free disk space for logs and data, and any modern operating system — Windows 10+, macOS 10.14+, or any current Linux distribution.

Downloading Apache Kafka

Go to the official Apache Kafka download page: kafka.apache.org/downloads. You will see several download options listed by version. Always download the latest stable release. Choose the binary download (not source) — it ends in .tgz (for Mac/Linux) or you can use the same file on Windows with a tool like 7-Zip or WinRAR to extract it.

As of 2024, the download file looks like: kafka_2.13-3.7.0.tgz. The "2.13" refers to the Scala version Kafka was compiled with. The "3.7.0" is the Kafka version number. Download the highest Kafka version number available.

DOWNLOAD PROCESS:

Visit kafka.apache.org/downloads
   ↓
Click the binary download link for latest version
   ↓
File: kafka_2.13-X.X.X.tgz (Mac/Linux) or .zip (Windows)
   ↓
Save to your home directory or a dedicated folder like /opt/kafka

Extracting and Organizing Kafka

On Mac or Linux, open a terminal and run:

tar -xzf kafka_2.13-3.7.0.tgz
mv kafka_2.13-3.7.0 ~/kafka
cd ~/kafka

On Windows, right-click the downloaded file and extract it using 7-Zip or the built-in extractor. Move the extracted folder to a simple path like C:\kafka to avoid spaces in the path (spaces in Windows paths cause problems with Kafka scripts).

Inside the Kafka Folder

After extraction, explore the folder structure. Understanding what is inside helps you navigate Kafka confidently.

kafka/
├── bin/               ← Shell scripts to start/stop Kafka, manage topics
│   ├── kafka-server-start.sh
│   ├── kafka-topics.sh
│   ├── kafka-console-producer.sh
│   ├── kafka-console-consumer.sh
│   └── zookeeper-server-start.sh
├── config/            ← Configuration files for brokers, ZooKeeper, etc.
│   ├── server.properties        (Kafka broker config)
│   ├── zookeeper.properties     (ZooKeeper config)
│   └── consumer.properties
├── libs/              ← Java library files Kafka needs to run
└── logs/              ← Where Kafka writes its own log files

Two Ways to Run Kafka Locally

There are two options for starting Kafka locally. The first uses ZooKeeper (the traditional approach). The second uses KRaft mode (the modern approach without ZooKeeper). This topic covers both. KRaft is the future of Kafka, but many existing systems still use ZooKeeper, so knowing both is valuable.

Option A: Running Kafka with ZooKeeper

In this mode, ZooKeeper runs as a separate process and manages cluster metadata for Kafka. You start ZooKeeper first, then start Kafka.

Step 1: Start ZooKeeper

Open a terminal window and run (from inside the kafka directory):

# Mac / Linux:
bin/zookeeper-server-start.sh config/zookeeper.properties

# Windows:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties

ZooKeeper starts and listens on port 2181 by default. You see a lot of log output. Keep this terminal window open — ZooKeeper must stay running.

Step 2: Start the Kafka Broker

Open a second terminal window and run:

# Mac / Linux:
bin/kafka-server-start.sh config/server.properties

# Windows:
bin\windows\kafka-server-start.bat config\server.properties

Kafka starts and listens on port 9092. You see log output ending with messages about the broker being ready. Keep this terminal open too.

PROCESS DIAGRAM - ZooKeeper Mode:

Terminal 1:             Terminal 2:
[ZooKeeper]      ←──── [Kafka Broker]
Port: 2181              Port: 9092

ZooKeeper manages cluster metadata.
Kafka broker handles producers and consumers.
Both must run simultaneously.

Option B: Running Kafka in KRaft Mode (No ZooKeeper)

KRaft mode eliminates ZooKeeper. Kafka manages its own cluster coordination. This is simpler to set up and is the direction all Kafka deployments are heading.

Step 1: Generate a Cluster UUID

# Mac / Linux:
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# Windows (PowerShell):
$KAFKA_CLUSTER_ID = bin\windows\kafka-storage.bat random-uuid

Step 2: Format the Storage Directory

# Mac / Linux:
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

# Windows:
bin\windows\kafka-storage.bat format -t %KAFKA_CLUSTER_ID% -c config\kraft\server.properties

Step 3: Start Kafka

# Mac / Linux:
bin/kafka-server-start.sh config/kraft/server.properties

# Windows:
bin\windows\kafka-server-start.bat config\kraft\server.properties

Kafka starts in KRaft mode. Only one terminal is needed. No ZooKeeper required.

PROCESS DIAGRAM - KRaft Mode:

Terminal 1 only:
[Kafka Broker (KRaft)]
Port: 9092

Kafka manages its own coordination internally.
Simpler setup. No separate ZooKeeper process.

Creating Your First Topic

With Kafka running, open a new terminal window and create a topic called "test-topic":

# Mac / Linux:
bin/kafka-topics.sh \
  --create \
  --topic test-topic \
  --partitions 1 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

# Windows:
bin\windows\kafka-topics.bat ^
  --create ^
  --topic test-topic ^
  --partitions 1 ^
  --replication-factor 1 ^
  --bootstrap-server localhost:9092

You see the message: Created topic test-topic.

Let's break down the flags:

--create: Tell Kafka to create a new topic.

--topic test-topic: Name the topic "test-topic."

--partitions 1: Create 1 partition (fine for local testing).

--replication-factor 1: Keep 1 copy of data (only 1 broker running locally, so replication-factor cannot exceed 1).

--bootstrap-server localhost:9092: Connect to Kafka at localhost on port 9092.

Listing All Topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Output: test-topic

Describing a Topic

bin/kafka-topics.sh --describe --topic test-topic --bootstrap-server localhost:9092

Output shows partition count, replication, and which broker is the leader:

Topic: test-topic  Partitions: 1  Replication factor: 1
  Topic: test-topic  Partition: 0  Leader: 0  Replicas: 0  Isr: 0

Producing Your First Messages

Open a new terminal and start the console producer — a simple command-line producer that reads your keyboard input and sends each line as a message:

bin/kafka-console-producer.sh \
  --topic test-topic \
  --bootstrap-server localhost:9092

A > prompt appears. Type messages and press Enter after each one:

> Hello Kafka!
> This is my first message.
> Learning Kafka is fun.

Each line you type becomes one message in the "test-topic" topic. Press Ctrl+C to stop the producer.

Consuming Messages

Open yet another terminal and start the console consumer to read the messages you just produced:

bin/kafka-console-consumer.sh \
  --topic test-topic \
  --from-beginning \
  --bootstrap-server localhost:9092

You see the messages appear:

Hello Kafka!
This is my first message.
Learning Kafka is fun.

The --from-beginning flag tells the consumer to start from the very first message in the topic (offset 0) rather than only reading new messages. Remove this flag if you only want to see messages produced after the consumer starts.

The Full Local Test Flow

WHAT JUST HAPPENED:

Terminal 1: [ZooKeeper or Kafka KRaft]  ← cluster coordinator
Terminal 2: [Kafka Broker]              ← stores messages
Terminal 3: [Console Producer]          ← you typed messages here
             "Hello Kafka!" → sent to topic: test-topic
Terminal 4: [Console Consumer]          ← read messages from topic
             received: "Hello Kafka!"

You just ran a complete producer → Kafka → consumer pipeline!

Using Docker to Run Kafka (The Easiest Option)

If you prefer not to manage processes manually, Docker is an excellent alternative. Docker runs Kafka inside a container — no Java installation needed, no manual process management, and you can start and stop the entire environment with one command.

With Docker and Docker Compose installed, create a file called docker-compose.yml with this content:

version: '3'
services:
  kafka:
    image: confluentinc/confluent-local:latest
    hostname: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"

Start it with:

docker-compose up -d

Kafka is now running. Use the same kafka-topics, kafka-console-producer, and kafka-console-consumer commands by pointing them at localhost:9092.

Common Setup Errors and How to Fix Them

Error: Port 9092 Already in Use

Another process is using port 9092. Stop the other process or change Kafka's port in config/server.properties by updating the line: listeners=PLAINTEXT://:9093

Error: Java Not Found

Java is not installed or the JAVA_HOME environment variable is not set. Install Java 11 or 17 and set the JAVA_HOME variable to point to your Java installation directory.

Error: ZooKeeper Connection Refused

ZooKeeper is not running or it failed to start. Check the ZooKeeper terminal for error messages. The most common cause is ZooKeeper's port 2181 being occupied by another process.

Error: Replication Factor Larger Than Available Brokers

When creating a topic with --replication-factor 3 on a local single-broker setup, Kafka rejects it. Use --replication-factor 1 for local testing.

Setting Up a Path Variable for Convenience

Running the full path to Kafka scripts every time is tedious. Add the Kafka bin directory to your system PATH so you can run kafka commands from anywhere.

On Mac/Linux, add this line to your ~/.bashrc or ~/.zshrc:

export PATH="$HOME/kafka/bin:$PATH"

Then reload the shell:

source ~/.bashrc

Now you can type kafka-topics.sh from any directory instead of the full path.

What You Have Achieved

You now have a working Kafka environment. You created a topic, produced messages, and consumed them. This is the foundation for every experiment in the rest of this course. Every concept — partitions, offsets, consumer groups, replication — becomes clearer when you can test it directly in your local environment.

Key Points

  • Kafka requires Java 8+. Install Java 11 or 17 LTS before setting up Kafka.
  • Download the binary release of Kafka from kafka.apache.org/downloads.
  • ZooKeeper mode requires two processes: ZooKeeper (port 2181) and Kafka broker (port 9092).
  • KRaft mode is simpler: no ZooKeeper, one process only. Format storage, then start Kafka.
  • Use kafka-topics.sh to create and describe topics. Use kafka-console-producer.sh to send messages. Use kafka-console-consumer.sh to read them.
  • Docker with docker-compose is the easiest way to get Kafka running quickly without manual process management.
Previous lessons
Back to courses
Next lessons

Leave a Comment