Event Hub Consumer Groups
Azure Event Hub is built for high-volume streaming. Multiple applications often need to read the same stream of events — a fraud detection service, a real-time dashboard, and an archiving system might all consume the same event stream simultaneously. Consumer groups make this possible without applications interfering with each other.
What Is a Consumer Group
A consumer group is an independent view of an Event Hub's event stream. Each consumer group maintains its own position (called an offset) in the stream. Applications reading through one consumer group do not affect the reading position of another consumer group.
Think of it like a television broadcast. One broadcast signal goes out, but ten different households can watch it independently. Each household starts watching when it wants, pauses when it wants, and rewinds when it wants. The broadcast does not change based on any one household's behavior.Event Hub (Stream of Events) [Event 1][Event 2][Event 3][Event 4][Event 5][Event 6][Event 7] Consumer Group A ($Default): Position: ---> reading at Event 6 Reader: Analytics Dashboard Consumer Group B (fraud-detection): Position: ---> reading at Event 4 Reader: Fraud Detection Service Consumer Group C (archive): Position: ---> reading at Event 2 Reader: Archive Service All three groups read from the SAME stream but each tracks its own position independently.
The Default Consumer Group
Every Event Hub comes with one consumer group named $Default. It exists automatically and cannot be deleted. For simple applications with only one reader, the default consumer group works fine.
Why Multiple Consumer Groups Matter
Without Consumer Groups
If two applications share a single cursor (offset), reading an event moves the cursor forward. The second application misses any events the first application already read.
PROBLEM: Shared Cursor
Stream: [E1][E2][E3][E4][E5]
^
Shared cursor position
App A reads E1 → cursor moves to E2
App B tries to read → gets E2 (missed E1!)
With Consumer Groups
Each application uses its own consumer group. Each group has its own cursor. Reading from one group does not move the cursor for another group.SOLUTION: Separate Consumer Groups Stream: [E1][E2][E3][E4][E5] Consumer Group A cursor: points to E1 → App A reads E1, cursor → E2 Consumer Group B cursor: still at E1 → App B reads E1 independently
Consumer Group Limits
| Tier | Max Consumer Groups per Event Hub |
|---|---|
| Basic | 1 (only $Default) |
| Standard | 20 |
| Premium | 100 |
| Dedicated | 1000 |
Basic tier restricts you to the default consumer group only. If you need multiple readers, use Standard or higher.
Creating a Consumer Group
Via Azure Portal
Azure Portal
→ Event Hub Namespace
→ Event Hub Instance
→ Consumer Groups
→ + Consumer Group
→ Enter Name: "fraud-detection"
→ Save
Via Azure CLI
az eventhubs eventhub consumer-group create \ --resource-group myResourceGroup \ --namespace-name myNamespace \ --eventhub-name myEventHub \ --name fraud-detection
Naming Rules
Consumer group names must start with a letter or number, be 1–50 characters long, and contain only letters, numbers, hyphens, underscores, and periods.
How a Consumer Group Tracks Position
Each consumer group tracks its reading position using an offset per partition. An offset is a byte position within a partition's log.Event Hub: 4 Partitions Partition 0: [E1-offset:0][E3-offset:250][E7-offset:510] Partition 1: [E2-offset:0][E5-offset:300][E8-offset:620] Partition 2: [E4-offset:0][E6-offset:190][E9-offset:440] Partition 3: [E10-offset:0] Consumer Group "analytics": Partition 0 → offset 250 (read up to E3) Partition 1 → offset 300 (read up to E5) Partition 2 → offset 440 (read up to E9) Partition 3 → offset 0 (read up to E10)
The Event Processor Host (or Azure SDK's EventProcessorClient) stores these offsets in Azure Blob Storage as checkpoints. If a consumer restarts, it picks up from the last checkpoint.
Consumer Group with EventProcessorClient
C# Example
var storageClient = new BlobContainerClient(
storageConnectionString, "checkpoints");
var processor = new EventProcessorClient(
storageClient,
"fraud-detection", // consumer group name
eventHubConnectionString,
"myEventHub");
processor.ProcessEventAsync += async (args) =>
{
var body = args.Data.Body.ToString();
Console.WriteLine($"Partition {args.Partition.PartitionId}: {body}");
// Checkpoint after processing
await args.UpdateCheckpointAsync();
};
processor.ProcessErrorAsync += (args) =>
{
Console.WriteLine($"Error: {args.Exception.Message}");
return Task.CompletedTask;
};
await processor.StartProcessingAsync();
The consumer group name "fraud-detection" is passed when constructing the EventProcessorClient. This ensures this application reads with its own independent cursor.
Real-World Architecture: Three Consumer Groups
IoT Devices
→ send telemetry events
|
v
+--------------------+
| Event Hub |
| "telemetry" |
| 4 Partitions |
+--------------------+
| | |
| | |
v v v
CG: $Default CG: ml-scoring CG: archive
| | |
v v v
Real-Time ML Anomaly Cold Storage
Dashboard Detection (Azure Data Lake)
(reads fast) (reads slower) (reads in batches)
Each consumer group processes at its own speed. The ML scoring service processes events slower than the dashboard but does not block the dashboard from reading ahead. The archive service runs in nightly batches but never misses an event because events are retained in Event Hub for the configured retention period (1–90 days depending on tier).
Consumer Group Best Practices
| Practice | Reason |
|---|---|
| Use a dedicated consumer group per application | Prevents offset conflicts between applications |
| Never share one consumer group across multiple unrelated apps | Causes missed events and race conditions |
| Use $Default only for testing or single-reader scenarios | Production workloads need isolation |
| Store checkpoints in Azure Blob Storage | Survives consumer restarts without data loss |
| Name consumer groups after the consuming application | Makes architecture diagrams self-documenting |
| Monitor consumer lag per consumer group | Detects slow consumers before they fall too far behind |
Monitoring Consumer Group Lag
Consumer lag is the difference between the newest event in a partition and the last checkpoint of a consumer group. High lag means the consumer is falling behind.Monitor lag using: Azure Portal → Event Hub → Consumer Groups → Offset ExplorerOr query it with Azure Monitor metrics filtering by consumer group name.