Service Bus Duplicate Detection

Azure Service Bus Duplicate Detection prevents the same message from being processed more than once. When enabled, Service Bus tracks the MessageId of every incoming message for a configurable time window. If a sender submits the same MessageId twice within that window, Service Bus silently discards the duplicate without delivering it to consumers.

Why Duplicate Messages Happen

Network issues, application retries, and timeout errors are the most common causes of duplicate messages. A sender sends a message and waits for an acknowledgment. The acknowledgment is lost due to a network timeout. The sender assumes the message was not delivered and sends it again. Service Bus has actually received both copies — and without duplicate detection, consumers process the same message twice.

WITHOUT Duplicate Detection:

Sender sends:  Message { MessageId: "order-101" }  --> Received by Service Bus
Network issue: ACK lost. Sender thinks it failed.
Sender retries: Message { MessageId: "order-101" }  --> Received AGAIN

Queue: [order-101][order-101]
Consumer A processes: order-101 (first)
Consumer B processes: order-101 (second -- DUPLICATE PROCESSING!)

WITH Duplicate Detection:

Sender sends:  Message { MessageId: "order-101" }  --> Accepted
Network issue: ACK lost. Sender retries.
Sender sends:  Message { MessageId: "order-101" }  --> REJECTED (duplicate)

Queue: [order-101]
Consumer processes order-101 exactly ONCE.

How Duplicate Detection Works

Service Bus maintains a history of MessageIds received:

History Window: 10 minutes

10:00: MessageId "order-101" --> ACCEPTED (new, not in history)
10:02: MessageId "order-102" --> ACCEPTED
10:03: MessageId "order-101" --> REJECTED (in history, within 10 min window)
10:11: MessageId "order-101" --> ACCEPTED (10 min window expired, treated as new)

Enable Duplicate Detection on a Queue

Azure Portal

1. Open the namespace and click + Queue
2. Enable the Duplicate Detection toggle
3. Set the Duplicate Detection History Time Window (default: 10 minutes)
4. Click Create

Duplicate detection cannot be enabled on an existing queue — create a new queue with it enabled from the start.

Azure CLI

az servicebus queue create \
  --resource-group rg-messaging-prod \
  --namespace-name myshopns \
  --name orders \
  --enable-duplicate-detection true \
  --duplicate-detection-history-time-window PT10M

Time Window Format (ISO 8601)

Enable Duplicate Detection on a Topic

az servicebus topic create \
  --resource-group rg-messaging-prod \
  --namespace-name myshopns \
  --name order-events \
  --enable-duplicate-detection true \
  --duplicate-detection-history-time-window PT30M

Setting MessageId in .NET SDK

For duplicate detection to work, every message must have a unique and consistent MessageId. The same business event must always use the same MessageId, so that retries are recognized as duplicates.

using Azure.Messaging.ServiceBus;

await using var client = new ServiceBusClient(connectionString);
await using var sender = client.CreateSender("orders");

string orderId = "order-101";

var message = new ServiceBusMessage(BinaryData.FromString($"{{ \"orderId\": \"{orderId}\" }}"))
{
    MessageId   = orderId,           // Use the business key as the MessageId
    Subject     = "NewOrder",
    ContentType = "application/json"
};

// First send
await sender.SendMessageAsync(message);
Console.WriteLine("Message sent (first attempt).");

// Simulated retry (e.g., after a timeout)
await sender.SendMessageAsync(message);
Console.WriteLine("Message sent (retry -- duplicate, will be silently discarded).");

Setting MessageId in Python SDK

from azure.servicebus import ServiceBusClient, ServiceBusMessage

connection_str = "Endpoint=sb://myshopns.servicebus.windows.net/;..."
queue_name = "orders"

order_id = "order-101"

with ServiceBusClient.from_connection_string(connection_str) as client:
    with client.get_queue_sender(queue_name) as sender:
        msg = ServiceBusMessage(
            body=f'{{"orderId": "{order_id}"}}',
            message_id=order_id       # Use business key as MessageId
        )

        # First attempt
        sender.send_messages(msg)
        print("Sent (first attempt).")

        # Retry
        sender.send_messages(msg)
        print("Sent again (will be deduplicated by Service Bus).")

Good vs Bad MessageId Design

Duplicate Detection History Window — Choosing the Right Value

Scenario: Payment system
  - Retry window for a payment: 5 minutes (max retry duration)
  - Set duplicate detection window to: 30 minutes (safe margin above retry window)

Scenario: Batch import system
  - Batch can be re-submitted up to 1 hour after first attempt
  - Set duplicate detection window to: 2 hours

Scenario: Daily unique events
  - Events are unique per day, same event same day = duplicate
  - Set duplicate detection window to: P1D (1 day)

Duplicate Detection on Partitioned Queues and Topics

Duplicate detection works across all partitions in partitioned queues and topics. Service Bus tracks MessageIds globally across all partitions within the detection window. This ensures duplicates are caught even when messages land on different partitions.

Duplicate Detection Limitations

Duplicate Detection vs Idempotent Consumers

Duplicate detection at the Service Bus level is not a substitute for idempotent consumer logic. If the consumer completes processing but crashes before calling CompleteMessageAsync(), Service Bus re-delivers the message. The consumer may process it again despite the message having a unique MessageId (because Service Bus deleted the message entry after first delivery). Always design consumers to handle duplicate processing gracefully.

Idempotent Consumer Pattern:

if (OrderAlreadyProcessed(orderId))
{
    // Skip processing — already done
    await receiver.CompleteMessageAsync(message);
    return;
}

ProcessOrder(orderId);
MarkOrderAsProcessed(orderId);
await receiver.CompleteMessageAsync(message);

Summary

Azure Service Bus Duplicate Detection prevents the same message from entering the queue or topic more than once within a configurable time window. It uses the MessageId property to identify duplicates. Senders must set a stable, business-meaningful MessageId — not a new GUID each time. Duplicate detection catches failures during send operations where the acknowledgment is lost and the sender retries. It requires Standard or Premium tier and must be enabled at queue or topic creation. Duplicate detection at the broker level complements but does not replace idempotent consumer logic in the application code.