Redis Pipelining

Pipelining sends many commands to Redis in one batch and reads all replies together. It removes the waiting time between commands. A task that takes a full second with single commands can finish in a few milliseconds with a pipeline.

The Shopping List Analogy

Imagine asking a helper to fetch ten items. One approach sends the helper to the store ten separate times, one item per trip. A better approach hands over a single list and lets the helper return once with everything. Pipelining is the single list.

Without pipelining               With pipelining
Client          Server           Client          Server
  │── SET a ───►  │                │── SET a ───►  │
  │◄── OK ──────  │                │── SET b ───►  │
  │── SET b ───►  │                │── SET c ───►  │
  │◄── OK ──────  │                │◄── OK ──────  │
  │── SET c ───►  │                │◄── OK ──────  │
  │◄── OK ──────  │                │◄── OK ──────  │
  3 round trips                    1 round trip

Why Round Trips Matter

Redis executes a command in microseconds. The network trip between client and server takes far longer, often a full millisecond or more. This travel time is called round-trip time, or RTT. Most of the delay in a command sequence comes from RTT, not from Redis itself.

Task: run 1,000 commands, RTT = 1 ms

Single commands:  1,000 × 1 ms  ≈ 1,000 ms
One pipeline:     1 × 1 ms + work ≈ 5 ms

Pipelining with redis-cli

The --pipe option reads raw commands from a file or another program and sends them in bulk.

$ cat commands.txt
SET user:1 "Asha"
SET user:2 "Ravi"
SET user:3 "Meena"

$ cat commands.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
errors: 0, replies: 3

Pipelining in Application Code

Most client libraries offer a pipeline object. You queue commands on it and call one execute method.

# Python example with the redis-py library
pipe = r.pipeline(transaction=False)
for i in range(1000):
    pipe.set("item:" + str(i), i)
results = pipe.execute()

The library sends all 1,000 commands together and returns a list of 1,000 replies in the same order.

Pipelining versus Transactions

Feature               Pipeline                  Transaction (MULTI/EXEC)
────────────────────  ────────────────────────  ─────────────────────────
Purpose               Reduce network waiting    Run commands as one unit
Other clients mix in  Yes, between commands     No, commands run together
Rolls back on error   No                        No, but runs in sequence
Speed gain            Large                     Small

Pipelines focus on speed. Transactions focus on grouped execution. Libraries often let you combine both, so one batch gains speed and runs as a unit.

Choosing a Batch Size

Redis holds every queued reply in memory until the pipeline completes. A pipeline of ten million commands uses a large amount of memory on both sides. Send commands in chunks of 500 to 10,000 and execute each chunk before building the next.

Practical Example – Loading Product Prices

Job: Load 50,000 product prices into Redis at start-up.

Slow way:  50,000 SET commands, one per trip
           Time: about 50 seconds at 1 ms RTT

Fast way:  50 pipelines of 1,000 SET commands
           Time: about 0.3 seconds

Data stored is identical. Only the waiting changes.

When Pipelining Does Not Help

• A later command needs the reply of an earlier command. Pipelines send everything before any reply arrives.
• The app sends only one command at a time.
• The client and server share the same machine, where RTT is almost zero.

Key Points

• Pipelining batches commands to cut round-trip waiting time.
• Redis returns replies in the order you sent the commands.
• redis-cli --pipe loads bulk data quickly.
• Pipelines do not guarantee grouped execution. Use MULTI and EXEC for that.
• Chunk large jobs to protect memory.