Kafka Python Client

Python offers a friendly way to work with Kafka, especially for data scripts, quick prototypes, and analytics jobs. The confluent-kafka package wraps the fast librdkafka library, so code stays short while performance stays high. Think of it as a simple remote control for a powerful machine: a few commands, and messages start moving.

Install the Library

pip install confluent-kafka

Use a virtual environment so the package stays isolated from other projects.

How the Pieces Fit

  Python script                         Kafka cluster
  +----------------+   produce()   +-----------------------+
  | Producer       | ------------> | topic: sensor-data    |
  +----------------+               |  P0 | P1 | P2         |
  +----------------+   poll()      |                       |
  | Consumer       | <------------ |                       |
  +----------------+               +-----------------------+

Write a Producer

from confluent_kafka import Producer

config = {"bootstrap.servers": "localhost:9092", "acks": "all"}
producer = Producer(config)

def report(err, msg):
    if err is not None:
        print("Delivery failed:", err)
    else:
        print("Delivered to", msg.topic(), "partition", msg.partition(),
              "offset", msg.offset())

for i in range(5):
    producer.produce("sensor-data",
                     key="sensor-1",
                     value="temperature=" + str(20 + i),
                     callback=report)
    producer.poll(0)

producer.flush()

The produce call queues a message and returns at once. The poll call serves delivery callbacks, and flush waits until every queued message reaches the broker. Forgetting flush at the end of a script often loses the last messages.

Write a Consumer

from confluent_kafka import Consumer

config = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "sensor-readers",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False
}
consumer = Consumer(config)
consumer.subscribe(["sensor-data"])

try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            continue
        if msg.error():
            print("Error:", msg.error())
            continue
        print(msg.key().decode(), "->", msg.value().decode())
        consumer.commit(message=msg)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

The poll call waits up to one second for a message. A return value of None means no data arrived in that time. Keys and values arrive as bytes, so decode them to text.

Send JSON Data

import json

event = {"sensor": "sensor-1", "temp": 22.5}
producer.produce("sensor-data",
                 key="sensor-1",
                 value=json.dumps(event).encode("utf-8"))
producer.flush()

On the consumer side, call json.loads on the decoded value to rebuild the dictionary.

Settings You Will Use Often

Setting	Side	Purpose
bootstrap.servers	Both	Entry addresses for the cluster
acks	Producer	Write confirmation level
linger.ms	Producer	Wait time to build larger batches
group.id	Consumer	Consumer group name
auto.offset.reset	Consumer	Start position without saved offsets

Handle Errors

• Check err in every delivery callback
• Check msg.error() before reading the value
• Catch exceptions around business logic and route failures to a dead letter topic
• Close the consumer in a finally block so the group rebalances promptly

Choosing Between Python Packages

The kafka-python package offers a pure Python approach that suits small learning projects. The confluent-kafka package delivers higher speed and receives frequent updates, which makes it the stronger choice for production work.

Practice Idea

Build a tiny weather pipeline. A producer script publishes a random temperature every second. A consumer script reads the stream, keeps a running average, and prints an alert when the value crosses 30 degrees. This small project covers keys, offsets, commits, and error handling in one place.

Run the Scripts

Start the consumer script first in one terminal, then run the producer script in a second terminal. The consumer prints each message as it arrives. Stop the consumer with Ctrl+C and start it again to confirm that it continues from the saved offset instead of replaying old data. This quick experiment shows how consumer groups and commits behave in real life.