RabbitMQ Work Queues Pattern

The work queues pattern (also called the task queue pattern) distributes time-consuming tasks among multiple worker processes. Instead of doing heavy work immediately when a request arrives, the system puts the task in a queue and lets one of several workers pick it up and process it. This keeps response times fast and spreads the workload evenly.

The Restaurant Kitchen Analogy

Imagine a busy restaurant. When a customer places an order, the waiter (producer) writes the order on a ticket and pins it to a rotating order board (the queue). Several cooks (workers/consumers) stand at the board. Each cook grabs the next available ticket and starts preparing that dish. No two cooks work on the same ticket. When a cook finishes, they reach for the next ticket. If one cook is slow with a complicated dish, the other cooks keep preparing their own orders. The restaurant serves many tables at once.

Work Queues Architecture

[Producer App]
     |
     | publishes tasks
     v
[Task Queue]
     |
     +----task1----> [Worker 1]  (processing)
     |
     +----task2----> [Worker 2]  (processing)
     |
     +----task3----> [Worker 3]  (processing)
     |
     [task4, task5 waiting...]

Each worker picks up one task at a time. RabbitMQ distributes tasks in round-robin order by default — Worker 1 gets task 1, Worker 2 gets task 2, Worker 3 gets task 3, then back to Worker 1 for task 4, and so on.

Why Not Process Tasks Inline

Consider a video processing website. A user uploads a video and clicks "Convert." If the server converts the video synchronously, the HTTP response takes minutes and the user sees a frozen browser. Worse, if 50 users upload at the same time, the server becomes overwhelmed.

With the work queue pattern, the server immediately responds "Upload received" and drops a task message into the queue. Background workers pick up conversion jobs one at a time. The user gets instant feedback, and the system scales by simply adding more worker processes.

Round-Robin vs Fair Dispatch

Round-Robin (Default)

RabbitMQ sends tasks to workers in strict rotation regardless of how busy each worker is. If Worker 1 gets a slow 10-minute task and Worker 2 is free, Worker 2 still waits for its turn before getting a new task. This is inefficient when tasks vary widely in processing time.

Fair Dispatch (Recommended)

Set prefetch_count=1 on each worker. This tells RabbitMQ: "Do not give me more than one task at a time until I acknowledge the current one." The next free worker immediately picks up the next queued task instead of waiting for round-robin rotation.

channel.basic_qos(prefetch_count=1)

With prefetch=1 (fair dispatch):

Worker 1: [slow task - running 10 min]
Worker 2: [done] <-- gets next task immediately
Worker 2: [done] <-- gets next task immediately
Worker 2: [done] <-- gets next task immediately

Worker 1 finishes and gets a new task only when ready.

Message Acknowledgement in Work Queues

Acknowledgements are critical in the work queue pattern. When a worker receives a task, it does not immediately tell RabbitMQ it is done. It waits until it has actually finished the task before sending an ack. If the worker crashes mid-task, RabbitMQ detects the missing ack and redelivers the task to another available worker.

def process_task(ch, method, properties, body):
    task = body.decode()
    print(f"Processing: {task}")
    do_heavy_work(task)          # actual processing
    ch.basic_ack(delivery_tag=method.delivery_tag)  # only then ack

Without acknowledgements, a crashed worker causes its task to be lost permanently.

Making Tasks Survive Broker Restart

Two settings ensure no task is lost if RabbitMQ restarts:

• Declare the task queue as durable: channel.queue_declare(queue='tasks', durable=True)
• Publish tasks as persistent: properties=pika.BasicProperties(delivery_mode=2)

Without both settings, all queued tasks vanish on a broker restart.

Scaling Workers

Adding more workers is the simplest way to scale task throughput. Each worker is an independent process that connects to RabbitMQ and consumes from the same queue. Start three workers on one server or spread them across multiple machines — RabbitMQ routes tasks to all of them automatically.

$ python worker.py &   # Worker 1
$ python worker.py &   # Worker 2
$ python worker.py &   # Worker 3

The task queue becomes the buffer that absorbs traffic spikes. During a spike, the queue grows. Workers drain it over time. The system never drops tasks, even under heavy load.

Real-World Examples

• Image resizing service: Users upload images. A queue holds resize tasks. Workers resize images in the background.
• Email sending platform: Thousands of marketing emails queue up. Multiple email workers send them concurrently without overloading the mail server.
• Report generation: Finance users request complex reports. Workers generate each report from the queue and store the result. Users download the report when ready.
• Data import pipeline: CSV files are uploaded and parsed into tasks. Workers import each row into the database without blocking the upload endpoint.

Complete Python Example

Producer (sends tasks):

import pika, sys

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks', durable=True)

task = ' '.join(sys.argv[1:]) or 'default task'
channel.basic_publish(
    exchange='',
    routing_key='tasks',
    body=task,
    properties=pika.BasicProperties(delivery_mode=2)
)
print(f"Sent: {task}")
connection.close()

Worker (processes tasks):

import pika, time

def callback(ch, method, props, body):
    task = body.decode()
    print(f"Working on: {task}")
    time.sleep(task.count('.'))  # simulate work
    print("Done")
    ch.basic_ack(delivery_tag=method.delivery_tag)

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='tasks', durable=True)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='tasks', on_message_callback=callback)
channel.start_consuming()

Summary

The work queues pattern offloads time-consuming tasks from producers to a pool of background workers. Set prefetch_count=1 for fair dispatch so fast workers do not idle while slow workers are busy. Use message acknowledgements so tasks are never lost on worker crashes. Declare the queue as durable and messages as persistent so tasks survive broker restarts. Scale simply by starting more worker processes — RabbitMQ distributes tasks automatically.