Kafka Serialization

Kafka stores and moves bytes. It knows nothing about strings, numbers, or customer records. Serialization converts an object into bytes before sending, and deserialization rebuilds the object after reading. Think of packing a suitcase: clothes (your object) get folded into a compact shape for the trip, and the traveler unpacks them at the destination in the same order.

Where Conversion Happens

  PRODUCER SIDE                              CONSUMER SIDE

  Java/Python object                         Java/Python object
        |                                           ^
        v                                           |
  Serializer                                  Deserializer
        |                                           ^
        v                                           |
     bytes  ---------- Kafka topic ---------->   bytes

Both sides must agree on the format. A producer that writes JSON and a consumer that expects numbers will fail.

Key and Value Use Separate Converters

Every message has a key and a value, and each one gets its own serializer. A typical setup uses a string key and a JSON value.

key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=com.example.JsonSerializer

Built-In Serializers

Type	Serializer Class
Text	StringSerializer
32-bit number	IntegerSerializer
64-bit number	LongSerializer
Decimal number	DoubleSerializer
Raw bytes	ByteArraySerializer
UUID	UUIDSerializer

Each class has a matching deserializer with the same prefix.

Common Data Formats

Format	Strength	Weakness
Plain text	Easy to read and debug	No structure
JSON	Human readable, flexible	Larger size, no enforced rules
Avro	Compact, schema enforced	Needs a schema registry
Protobuf	Fast, strong typing	Needs generated classes

JSON suits learning and small projects. Avro and Protobuf suit large systems where many teams share data and formats change over time.

Build a Custom Serializer

Implement the Serializer interface to convert your own class.

public class OrderSerializer implements Serializer<Order> {
    private final ObjectMapper mapper = new ObjectMapper();

    @Override
    public byte[] serialize(String topic, Order order) {
        try {
            return mapper.writeValueAsBytes(order);
        } catch (Exception e) {
            throw new SerializationException("Cannot write order", e);
        }
    }
}

The matching deserializer reads the bytes with mapper.readValue and returns an Order object.

What Happens When Formats Change

Business needs evolve. A new field appears, an old field disappears, or a type changes. Consumers running older code must keep working during the transition.

Safe Changes

• Add an optional field with a default value
• Remove an optional field

Risky Changes

• Rename a field
• Change a field type, such as number to text
• Remove a required field

A Schema Registry checks each new schema against compatibility rules and blocks changes that would break readers.

Message Size and Compression

Compact formats cut network and storage cost. A record with ten fields in JSON may use twice the space of the same record in Avro. Add producer compression, such as lz4 or zstd, for a further reduction.

Debugging Tips

• A garbled string on the console usually means a binary format read as text
• A SerializationException on the consumer points to a mismatch between writer and reader
• Keep a sample message from each topic in the documentation

Choose Early

Pick one format per topic at design time and document it. Changing the format of a live topic forces every consumer to upgrade in lockstep, which becomes expensive as systems grow.

Quick Example of the Idea

Take the order record with the ID 101 and the item Laptop. A JSON serializer writes the text {"id":101,"item":"Laptop"} and turns each character into a byte. A consumer reads those bytes, decodes the text, and rebuilds an order with the same two fields. Any mismatch in field names or types breaks the rebuild, which is why teams share schemas and test them.