Redis Job Queues
A job queue lets a web app hand slow work to background workers. Sending an email, resizing an image, or building a report can take seconds. The web page answers the visitor at once and drops a job into a queue. Workers pick up jobs and finish them without delay for the visitor. Redis Lists make a simple and fast queue.
The Restaurant Kitchen Analogy
A waiter takes your order and clips a ticket to the kitchen rail. Cooks grab tickets one by one. The waiter returns to other tables without waiting for the food. The ticket rail is the queue, the waiter is the producer, and the cooks are workers.
Producers (web servers) Queue (Redis List) Workers ┌────────────┐ ┌──────────────────┐ ┌──────────┐ │ Web 1 │ ─ LPUSH ──────► │ job5 job4 job3.. │ ─────►│ Worker A │ │ Web 2 │ ─ LPUSH ──────► │ │ ─────►│ Worker B │ └────────────┘ └──────────────────┘ └──────────┘
The Basic Queue: LPUSH and BRPOP
The producer pushes jobs onto the left end of a List. Workers pop jobs from the right end, so the oldest job leaves first.
127.0.0.1:6379> LPUSH jobs '{"type":"email","to":"asha@example.com"}'
(integer) 1
127.0.0.1:6379> LPUSH jobs '{"type":"resize","image":"photo7.jpg"}'
(integer) 2
127.0.0.1:6379> BRPOP jobs 30
1) "jobs"
2) "{\"type\":\"email\",\"to\":\"asha@example.com\"}"
BRPOP is a blocking pop. The worker waits up to 30 seconds for a job to arrive, and uses no CPU while waiting. A timeout of 0 waits forever.
The Danger of Losing Jobs
With BRPOP, a job leaves the queue the moment a worker takes it. A crash in the middle of the work loses the job for good.
Queue: [job3] [job2] [job1] Worker pops job1 → job1 exists only in the worker's memory Worker crashes → job1 vanishes
A Safer Queue: BLMOVE
BLMOVE moves a job from the waiting list into a processing list in one atomic step. The job always lives in one list or the other.
127.0.0.1:6379> BLMOVE jobs jobs:processing RIGHT LEFT 30
"{\"type\":\"resize\",\"image\":\"photo7.jpg\"}"
When the worker finishes, it removes the job from the processing list.
127.0.0.1:6379> LREM jobs:processing 1 '{"type":"resize","image":"photo7.jpg"}'
(integer) 1
jobs (waiting) ──BLMOVE──► jobs:processing ──LREM (done)──► gone
│
└── worker crashed? A recovery task
moves old entries back to "jobs"
A small recovery script checks the processing list and returns stuck jobs to the waiting list.
Delayed Jobs with a Sorted Set
Use a Sorted Set when a job must run later. The score holds the run time.
127.0.0.1:6379> ZADD jobs:delayed 1718010000 '{"type":"reminder","user":1001}'
(integer) 1
127.0.0.1:6379> ZRANGEBYSCORE jobs:delayed 0 1718009999 LIMIT 0 10
(empty array)
A scheduler loop asks for jobs whose time has arrived, moves them into the main queue, and removes them from the Sorted Set.
Retries and Failed Jobs
- Include a retry counter inside each job.
- Re-queue a failed job with a delay that grows after each failure.
- Move a job to a failed list after the final attempt, so people can inspect it.
Queue Tools Built on Redis
Mature libraries handle retries, scheduling, and monitoring for you. Popular choices include BullMQ for Node.js, Celery for Python, Sidekiq for Ruby, and Hangfire for .NET. Redis Streams offer another queue style with consumer groups and acknowledgments for larger systems.
Practical Example – Welcome Email
1. A visitor signs up on the website. 2. The server saves the account, then runs LPUSH jobs with an email job. 3. The server answers "Account created" within milliseconds. 4. A worker runs BLMOVE, sends the email, and runs LREM to confirm. 5. A failed send returns to the queue with retry count 1.
Key Points
- LPUSH adds jobs and BRPOP lets workers wait for them.
- BLMOVE keeps a job safe in a processing list until the work completes.
- A Sorted Set schedules delayed jobs.
- Retry counters and failed lists make queues dependable.
- Ready-made libraries save effort on real projects.