Redis Streams

A Redis Stream is an append-only log of events. Each event has a unique ID, a timestamp, and a set of field-value pairs. Unlike Pub/Sub, Streams store messages even when no one is listening, allow multiple consumer groups to process the same events independently, and let consumers acknowledge what they processed.

The Package Tracking Log Analogy

A courier company keeps a log book. Every time a package changes status, a worker writes a new line in the log — the time, the package ID, and what happened. Nothing in the log is ever erased. Multiple departments (billing, customer service, analytics) all read from the same log at their own pace without interfering with each other. Redis Streams work exactly like this log book.

Redis Stream: "shipments"

ID              │ event        │ packageId │ location
────────────────┼──────────────┼───────────┼────────────
1710000001-0    │ "pickup"     │ "PKG-001" │ "Mumbai"
1710000050-0    │ "in-transit" │ "PKG-001" │ "Pune"
1710000120-0    │ "delivered"  │ "PKG-001" │ "Bengaluru"
1710000200-0    │ "pickup"     │ "PKG-002" │ "Delhi"
         ▲
         └── auto-generated ID: Unix milliseconds + sequence number

Adding Events to a Stream with XADD

127.0.0.1:6379> XADD shipments * event pickup packageId PKG-001 location Mumbai
"1710000001-0"   ← auto-generated ID (use * to auto-generate)

127.0.0.1:6379> XADD shipments * event in-transit packageId PKG-001 location Pune
"1710000050-0"

127.0.0.1:6379> XADD shipments * event delivered packageId PKG-001 location Bengaluru
"1710000120-0"

Reading from a Stream

XRANGE – Read Events in Order

Read all events from beginning:
  XRANGE shipments - +
  1) 1) "1710000001-0"
     2) 1) "event"   2) "pickup"   3) "packageId"   4) "PKG-001" ...
  2) 1) "1710000050-0"
     2) 1) "event"   2) "in-transit" ...
  3) ...

Read only the first 2 events:
  XRANGE shipments - + COUNT 2

Read events after a specific ID:
  XRANGE shipments 1710000050-0 +

XLEN – Count Events in the Stream

127.0.0.1:6379> XLEN shipments
(integer) 3

Reading New Events as They Arrive with XREAD

Non-blocking read (get events newer than ID 1710000001-0):
  XREAD COUNT 10 STREAMS shipments 1710000001-0

Blocking read (wait up to 5 seconds for new events):
  XREAD COUNT 10 BLOCK 5000 STREAMS shipments $
  ($ means "only new events published after this command starts")

Consumer Groups – Multiple Workers Processing the Same Stream

A consumer group lets several workers share the work of processing a stream. Each event goes to exactly one worker in the group. Other groups process the same stream independently, getting their own copy of every event.

Stream: "orders"

          ┌──────────────────────────────────────────┐
          │  Consumer Group: "billing"               │
          │                                          │
          │  Worker A  ←── order-001                 │
          │  Worker B  ←── order-002                 │
          │  Worker C  ←── order-003                 │
          └──────────────────────────────────────────┘

          ┌──────────────────────────────────────────┐
          │  Consumer Group: "notifications"         │
          │                                          │
          │  Worker X  ←── order-001 (same events,   │
          │  Worker Y  ←── order-002  separate copy) │
          └──────────────────────────────────────────┘

Each group processes independently. Billing does not block notifications.

Creating a Consumer Group

Create group "billing" starting from the beginning of the stream:
  XGROUP CREATE orders billing 0

Create group starting from new events only:
  XGROUP CREATE orders notifications $

Workers Reading from a Group with XREADGROUP

Worker A reads up to 5 undelivered events:
  XREADGROUP GROUP billing workerA COUNT 5 STREAMS orders >

  The > symbol means "give me undelivered messages"

Returns events assigned to workerA.
Redis marks them as "pending" for this worker.

Acknowledging Processed Events with XACK

After processing event "1710000001-0":
  XACK orders billing 1710000001-0
  → (integer) 1

Redis removes it from the pending list for this worker.
If the worker crashes without acknowledging, Redis keeps the
event as pending so another worker can reclaim it.

Pending Events – Reclaiming After a Crash

View pending events for a group:
  XPENDING orders billing - + 10

If workerA crashed and held pending events, workerB can claim them:
  XCLAIM orders billing workerB 60000 1710000001-0
  (60000 ms = reclaim events idle for more than 1 minute)

Stream vs Pub/Sub vs List Queue

┌─────────────────────┬────────┬──────────┬────────────────┐
│  Feature            │  List  │  Pub/Sub │  Stream        │
├─────────────────────┼────────┼──────────┼────────────────┤
│  Stores messages    │  Yes   │  No      │  Yes           │
│  Replay old events  │  No    │  No      │  Yes           │
│  Multiple groups    │  No    │  Yes     │  Yes           │
│  Acknowledgements   │  No    │  No      │  Yes           │
│  Ordered by time    │  Yes   │  N/A     │  Yes           │
└─────────────────────┴────────┴──────────┴────────────────┘

Key Points

  • Streams store events permanently in time order. Old events remain unless you trim the stream.
  • XADD appends a new event; XRANGE reads events by ID range; XLEN counts events.
  • Consumer groups let multiple independent workers share a stream, each getting a unique subset of events.
  • XACK marks an event as processed. Unacknowledged events stay pending so another worker can reclaim them after a failure.
  • Use Streams when you need durability, replay, and multi-group delivery — things that Pub/Sub and Lists cannot provide.
Previous lessons
Back to courses
Next lessons

Leave a Comment