Kafka Java Producer

The Java client library is the official way to write producer code for Kafka. A few lines of code create a producer, build a message, and send it to a topic. Think of the producer as a courier: it receives parcels from your program, groups them into batches, and delivers them to the correct broker.

Add the Dependency

Add the client library to a Maven project.

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.7.0</version>
</dependency>

Use the version that matches your cluster or a recent stable release.

The Send Path

  Your code
     |  send(record)
     v
  Serializer  -->  Partitioner  -->  Batch buffer  -->  Sender thread  -->  Broker

The serializer converts keys and values to bytes. The partitioner picks a partition from the key. The buffer collects records into batches, and a background thread ships each batch to the leader broker.

A Complete Producer

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class OrderProducer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("key.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer",
            "org.apache.kafka.common.serialization.StringSerializer");
        props.put("acks", "all");

        try (KafkaProducer<String, String> producer =
                 new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record =
                new ProducerRecord<>("orders", "order-101", "Laptop x1");
            producer.send(record);
        }
    }
}

The try-with-resources block closes the producer automatically, and close flushes any buffered records before the program ends.

Understand the Settings

Setting	Purpose
bootstrap.servers	First addresses used to find the cluster
key.serializer	Class that turns the key into bytes
value.serializer	Class that turns the value into bytes
acks	How many brokers confirm each write

Check the Result with a Callback

The send method works asynchronously and returns immediately. Attach a callback to learn what happened.

producer.send(record, (metadata, exception) -> {
    if (exception != null) {
        System.err.println("Send failed: " + exception.getMessage());
    } else {
        System.out.println("Saved in partition " + metadata.partition()
            + " at offset " + metadata.offset());
    }
});

The callback runs on the producer thread, so keep it short. Heavy work inside the callback slows every later send.

Synchronous Sending

Call get on the returned future to wait for the result.

RecordMetadata meta = producer.send(record).get();

Waiting for each record limits throughput sharply. Choose this style only for rare, critical messages.

Tune Batching

props.put("linger.ms", "10");
props.put("batch.size", "32768");
props.put("compression.type", "lz4");

A linger of 10 milliseconds lets the producer gather more records per batch. Larger batches and compression raise throughput and reduce network cost.

Safe Production Settings

props.put("acks", "all");
props.put("enable.idempotence", "true");
props.put("retries", Integer.MAX_VALUE);

These three lines protect against lost and duplicated messages during broker failures.

Reuse One Producer

A KafkaProducer is thread-safe. Create one instance per application and share it across threads. Creating a new producer for every message wastes connections and slows the application badly.

Flush and Close

Call flush to push buffered records to the brokers on demand. Call close during shutdown. Skipping both can silently drop the last few messages.

Message Headers and Partitions

A record can carry headers, which are small labels such as a trace ID or a source system name. Headers travel beside the key and value without changing the payload. A record can also name its partition directly through an extra constructor argument. Most programs skip this and let the key decide, because key-based placement keeps related messages together and spreads load evenly.

Typical Beginner Mistakes

• Forgetting to close or flush the producer before the program exits
• Creating a new producer for each message
• Ignoring the callback and missing failed sends
• Sending null keys when ordering matters