Kafka Dead Letter Topics

Every real pipeline meets bad data and temporary failures. A malformed message, a database outage, or a network drop can stop a consumer cold. Strong error handling keeps the pipeline moving while preserving every failed message for later review. Picture a mail sorting center: letters with unreadable addresses go into a special bin for staff to inspect, and the belt keeps running for everything else.

Two Kinds of Failures

Type	Example	Right Response
Temporary	Database timeout, network blip	Retry after a short delay
Permanent	Broken JSON, missing required field	Move aside and continue

Retrying a permanent failure wastes time and blocks every message behind it. A message that keeps failing and blocks a partition is called a poison pill.

Producer Side Errors

Producers retry retriable errors automatically. Two settings control the behavior.

retries=2147483647
delivery.timeout.ms=120000
enable.idempotence=true

The delivery timeout caps the total time spent on one send, including all retries. Idempotence prevents retries from writing duplicates. Non-retriable errors, such as a message that exceeds the size limit, arrive in the send callback, where the application decides what to do.

Consumer Side Errors

Wrap message processing in a try and catch block. Never let one bad record crash the poll loop.

for (ConsumerRecord<String, String> record : records) {
    try {
        process(record);
    } catch (TemporaryException e) {
        retryLater(record);
    } catch (Exception e) {
        sendToDeadLetterTopic(record, e);
    }
}
consumer.commitSync();

The commit happens after the loop, so failed records have a destination before the offset moves forward.

The Dead Letter Topic Pattern

A dead letter topic, often shortened to DLT, stores messages that processing could not handle. The main consumer writes the failed message there and moves on.

  orders  --->  [ Consumer ]  --->  success  --->  database
                     |
                     | failure
                     v
              orders.dlt   --->  review tool or repair job

Name the topic after its source with a dlt suffix. A clear name lets support teams find failures quickly.

What to Store in a Dead Letter Message

• The original key and value, unchanged
• The source topic, partition, and offset
• The error message and exception type
• The time of failure
• The retry count so far

Place this extra information in message headers. The original payload stays untouched, which makes later replay simple.

Retry Topics with Delays

Blocking a partition while waiting for a slow database harms throughput. A retry topic chain solves this by moving the message to a delayed lane.

  orders ---> orders.retry.1m ---> orders.retry.10m ---> orders.dlt
              (wait 1 minute)      (wait 10 minutes)     (final stop)

Each retry consumer pauses until the message is old enough, then tries again. After the last stage fails, the message lands in the dead letter topic. Main traffic never waits.

Replay After the Fix

Fix the bug or restore the dependency, then run a small replay job. The job reads the dead letter topic and publishes each message back to the source topic. Add a header that marks replayed messages so downstream systems can recognize them.

Deserialization Errors

A record that cannot be decoded throws an error before application code sees it. Wrap the deserializer with an error-handling version that returns a marker object instead of crashing. The consumer then routes the raw bytes to the dead letter topic.

Best Practices

• Separate temporary and permanent errors in code
• Limit retries to a fixed number
• Monitor the size of every dead letter topic and alert on growth
• Set long retention on dead letter topics so engineers have time to investigate
• Remove sensitive data or protect the topic with access rules