Event Hub with Apache Kafka Protocol

Azure Event Hub supports the Apache Kafka protocol natively. Applications written for Apache Kafka can connect to Event Hub without changing their code — just by updating the connection configuration. This makes Event Hub a drop-in replacement for a self-managed Kafka cluster, removing the need to deploy, patch, scale, and operate Kafka infrastructure yourself.

What Apache Kafka Is

Apache Kafka is an open-source distributed event streaming platform widely used for building real-time data pipelines. It uses a producer-consumer model with topics and partitions — concepts that map directly to Event Hub.

Many organizations already run applications built on the Kafka API. These applications can connect to Azure Event Hub using the same Kafka client libraries they already use.

Concept Mapping: Kafka to Event Hub

+------------------------+------------------------+
|    Apache Kafka        |    Azure Event Hub     |
+------------------------+------------------------+
| Kafka Cluster          | Event Hub Namespace    |
| Kafka Topic            | Event Hub              |
| Partition              | Partition              |
| Producer               | Producer               |
| Consumer Group         | Consumer Group         |
| Offset                 | Offset                 |
| Broker                 | Managed Service        |
| ZooKeeper              | Not needed (managed)   |
+------------------------+------------------------+

The parallel is nearly complete. A Kafka application sees Event Hub as a Kafka broker. Partitions, consumer groups, and offsets work identically.

Why Use Event Hub Instead of Running Kafka

Aspect	Self-Managed Kafka	Azure Event Hub (Kafka-compatible)
Infrastructure	You provision VMs, storage, networking	Fully managed by Azure
ZooKeeper	Required for cluster coordination	Not needed
Scaling	Manual broker additions	Automatic scaling
Patching	Your team patches OS and Kafka	Azure handles all patches
Geo-replication	Complex MirrorMaker setup	Geo-DR built in
Cost model	Always-on VMs	Pay per throughput unit
Kafka API compatibility	Full native	Kafka 1.0+ protocols

Configuring a Kafka Application to Use Event Hub

The only change needed in a Kafka application is its bootstrap server and authentication configuration. No code changes are required.

Standard Kafka Configuration

bootstrap.servers=localhost:9092
security.protocol=PLAINTEXT

Updated Configuration for Event Hub

bootstrap.servers=myNamespace.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
  required username="$ConnectionString" \
  password="Endpoint=sb://myNamespace.servicebus.windows.net/;
            SharedAccessKeyName=RootManageSharedAccessKey;
            SharedAccessKey=...";

Port 9093 is the Kafka-over-TLS port that Event Hub listens on. Authentication uses a connection string as the SASL password.

Python Producer Example (no code change from Kafka)

from confluent_kafka import Producer
config = {
    'bootstrap.servers': 'myNamespace.servicebus.windows.net:9093',
    'security.protocol': 'SASL_SSL',
    'sasl.mechanism': 'PLAIN',
    'sasl.username': '$ConnectionString',
    'sasl.password': 'Endpoint=sb://myNamespace.servicebus.windows.net/;
                      SharedAccessKeyName=RootManageSharedAccessKey;
                      SharedAccessKey=your-key-here'
}
producer = Producer(config)
producer.produce(
    topic='orders',          # This is an Event Hub named "orders"
    key='customer-101',      # Partition key
    value='{"orderId": "ORD-001", "amount": 99.99}'
)
producer.flush()

Java Consumer Example

Properties props = new Properties();
props.put("bootstrap.servers", "myNamespace.servicebus.windows.net:9093");
props.put("security.protocol", "SASL_SSL");
props.put("sasl.mechanism", "PLAIN");
props.put("sasl.jaas.config",
    "org.apache.kafka.common.security.plain.PlainLoginModule required "
    + "username=\"$ConnectionString\" "
    + "password=\"Endpoint=sb://myNamespace...\";");
props.put("group.id", "my-consumer-group");   // Consumer Group in Event Hub
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("auto.offset.reset", "earliest");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("orders")); // Event Hub name
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        System.out.printf("Partition: %d | Offset: %d | Value: %s%n",
            record.partition(), record.offset(), record.value());
    }
}

Kafka Protocol Support Tier Requirements

Tier	Kafka Protocol Support
Basic	No
Standard	Yes
Premium	Yes
Dedicated	Yes

Basic tier does not support the Kafka protocol. Upgrade to Standard or higher.

How Kafka Concepts Map in Practice

Kafka Topics = Event Hubs

Kafka application references topic "orders"
→ Event Hub must have an Event Hub instance named "orders"
→ Create it manually before the Kafka app connects
→ Set partition count when creating (cannot change later on Standard)

Consumer Groups

Kafka consumer group names map to Event Hub consumer groups. The Kafka application's group.id configuration becomes the consumer group name in Event Hub.

Kafka config: group.id=fraud-detection
→ Creates consumer group "fraud-detection" in Event Hub automatically
   (on first connection, if the group does not already exist)

Offsets

Kafka stores consumer offsets in the special __consumer_offsets internal topic. Event Hub stores offsets natively. The Kafka client's offset commit API works transparently.

Kafka MirrorMaker 2 for Migration

Teams running an existing self-managed Kafka cluster can migrate to Event Hub gradually using Kafka MirrorMaker 2. MirrorMaker replicates events from the source Kafka cluster to Event Hub in real time.

Migration Path:
Phase 1 – Replicate:
  Self-Managed Kafka → MirrorMaker 2 → Azure Event Hub
Phase 2 – Dual-read:
  Consumers gradually moved to read from Event Hub
  Old consumers still read from Kafka
Phase 3 – Switch producers:
  Producers updated to publish directly to Event Hub
  MirrorMaker stopped
Phase 4 – Decommission:
  Self-managed Kafka cluster retired

Kafka API Limitations in Event Hub

Feature	Status in Event Hub
Producer and Consumer APIs	Fully supported
Consumer Groups	Fully supported
Kafka Streams	Not supported
Kafka Connect	Supported (via connectors)
Kafka Transactions	Not supported
Admin API (topic creation via Kafka)	Limited support
Log compaction	Not supported

For advanced Kafka features like Streams or Transactions, Event Hub is not a complete replacement. Evaluate these requirements before migrating.

Performance and Throughput

Event Hub Throughput Units (Standard Tier):
  1 TU = 1 MB/s ingress, 2 MB/s egress, 1000 events/second
Kafka application sending at 5 MB/s → needs at least 5 TUs configured
Premium and Dedicated tiers:
  Processing Units (PU) replace Throughput Units
  Higher limits with reserved capacity

Set throughput units in: Event Hub Namespace → Scale → Throughput Units

Enable Auto-Inflate to let the namespace scale throughput units up automatically when demand spikes.

Summary

Azure Event Hub's native Kafka protocol support lets existing Kafka applications connect with only a configuration change — no code modifications required. Kafka producers, consumers, and consumer groups all work identically. Event Hub replaces the need to manage Kafka brokers, ZooKeeper, and infrastructure scaling. Kafka MirrorMaker 2 enables gradual migration from self-managed Kafka to Event Hub. Kafka Streams and Transactions are not supported, so evaluate those requirements before choosing Event Hub as a Kafka replacement.

Previous lesson

Back to course

Next lesson