SD Message Queues and Async Processing

In synchronous communication, the caller waits for the response before doing anything else. In asynchronous processing, the caller sends a request and immediately moves on to other work — the response or result comes later. Message queues are the infrastructure that makes async processing possible in distributed systems.

Think of a post office. Sending a letter (async) does not require both the sender and recipient to be available at the same time. The post office (message queue) holds the letter and delivers it when the recipient is ready. This contrasts with a phone call (sync) where both parties must be available simultaneously.

Why Message Queues Are Needed

Consider an e-commerce order placement flow without a message queue:

User clicks "Place Order"
    ↓
System must:
1. Save order to database       (50ms)
2. Charge payment               (300ms)
3. Send confirmation email      (200ms)
4. Update inventory             (100ms)
5. Notify warehouse             (150ms)
6. Generate invoice             (200ms)
Total: ~1 second before user gets a response

Problem: User waits 1 second. Any step failure rolls back everything.

With a message queue:

User clicks "Place Order"
    ↓
System:
1. Saves order + publishes "OrderPlaced" message   (50ms)
→ Returns "Order confirmed!" to user immediately

Background workers handle the rest asynchronously:
Worker 1: Charges payment
Worker 2: Sends confirmation email
Worker 3: Updates inventory
Worker 4: Notifies warehouse
Worker 5: Generates invoice

User gets instant response. All other tasks happen in the background.

Key Concepts in Message Queues

Producer

The producer creates and sends messages to the queue. It does not know or care who processes the message or when.

Consumer (Worker)

The consumer reads messages from the queue and processes them. Multiple consumers can process messages concurrently.

Message Queue / Broker

The broker sits between producers and consumers, storing messages until they are consumed. It decouples the producer from the consumer.

Message

A structured piece of data passed between producer and consumer. Usually JSON.

+----------+         +-------------+         +----------+
|          | Publish |             | Consume  |          |
| Producer | ------> |   Message   | -------> | Consumer |
| (App)    |         |    Queue    |          | (Worker) |
+----------+         +-------------+         +----------+
                      Holds message
                      until consumed

Message Queue Models

Point-to-Point (Queue Model)

Each message gets consumed by exactly one consumer. Multiple consumers compete for messages — whoever picks it first processes it. This enables horizontal scaling of workers.

Producer → Queue → Consumer A processes msg 1
                 → Consumer B processes msg 2
                 → Consumer C processes msg 3

Each message consumed once. Order processing queue follows this model.

Publish-Subscribe (Pub/Sub Model)

A message gets delivered to all subscribers simultaneously. One event notifies many different services at once.

Producer publishes "UserRegistered" event
        ↓
     Message Broker (Topic: user_registered)
        ↓
  +-----------+-------------+------------------+
  |           |             |                  |
Consumer 1: Consumer 2:  Consumer 3:
Send welcome  Create user  Set up free
  email       profile      trial account

Message Delivery Guarantees

Different systems offer different guarantees about message delivery. Understanding these is critical for building reliable systems.

At-Most-Once Delivery

The message is delivered zero or one time. The producer sends once and does not retry. If the consumer crashes before processing, the message is lost. Fastest but least reliable.

Producer → Queue → Consumer crashes → Message lost! (never redelivered)
Best for: Metrics, non-critical analytics where some loss is acceptable

At-Least-Once Delivery

The message is delivered one or more times. The queue keeps the message until the consumer acknowledges it. If the consumer crashes before acknowledging, the message redelivers. Risk of duplicates.

Producer → Queue → Consumer processes → Consumer crashes before ACK
→ Queue redelivers → Consumer processes again (duplicate!)

Best for: Emails, order processing (need idempotent consumers)

Exactly-Once Delivery

The message is delivered precisely once, even in failure scenarios. The most reliable but most complex and resource-intensive mode.

Best for: Financial transactions, inventory deductions
Achieved by: Transactions, idempotency keys, deduplication logic

Message Acknowledgment (ACK)

After consuming a message, the consumer sends an acknowledgment to the queue. The queue only deletes the message after receiving the ACK. This prevents message loss if a consumer crashes mid-processing.

Queue:    Sends message to Consumer
Consumer: Starts processing...
Consumer: Successfully processed
Consumer: Sends ACK to queue
Queue:    Deletes the message

If consumer crashes before ACK:
Queue:    No ACK received → Returns message to queue → Re-delivers to another consumer

Dead Letter Queue (DLQ)

Sometimes a message cannot be processed — bad format, invalid data, or repeated consumer failures. Instead of blocking the queue or losing the message, it moves to a Dead Letter Queue (DLQ) for manual inspection or special handling.

Normal Queue:
Message → Consumer fails → Retry (3 times)
          Still fails → Move to Dead Letter Queue

Dead Letter Queue:
→ Alert team
→ Manual inspection
→ Fix and re-process, or discard

Message Queue vs Direct Service-to-Service Call

Task Queues for Background Jobs

A task queue is a specific type of message queue used for background job processing. Instead of running a long task in the web request thread, the application creates a task and a worker handles it asynchronously.

Without task queue:
User uploads 100MB video
→ Server starts processing (resize, transcode)
→ User waits 5 minutes for response → Times out!

With task queue:
User uploads 100MB video
→ Server saves video, creates task "process_video:abc123"
→ Returns "Upload received! Processing in background"
→ Worker picks up task, processes over next 5 minutes
→ User gets notified when ready

Tools: Celery (Python), Bull (Node.js), Sidekiq (Ruby)

Popular Message Queue Systems

Apache Kafka Deep Dive

Kafka is designed for extremely high-throughput event streaming. Unlike traditional queues where messages disappear after consumption, Kafka retains messages for a configurable period, allowing multiple consumer groups to replay and process the same stream.

Kafka Architecture:

Topic: "order_events"
  Partition 0: [msg1, msg2, msg3, msg4, msg5...]
  Partition 1: [msg6, msg7, msg8, msg9...]
  Partition 2: [msg10, msg11, msg12...]

Consumer Group A (Order Service): reads from partition 0, 1, 2
Consumer Group B (Analytics):     reads from same partitions independently

Both groups process the same events without interfering with each other.
Kafka retains events for 7 days (configurable) → replay is possible.

Use case: LinkedIn uses Kafka to process over 7 trillion messages per day — activity events, metrics, and real-time analytics all flow through Kafka.

Summary

Message queues transform tightly coupled synchronous systems into loosely coupled, resilient architectures. Producers and consumers operate independently, failures are contained, and workloads scale horizontally by adding more workers. Understanding delivery guarantees, ACKs, and DLQs is essential for building reliable async workflows. Kafka handles massive scale with event streaming, while RabbitMQ and SQS handle traditional task queue patterns.