Redis HyperLogLog

HyperLogLog is a special Redis structure that counts unique items while using a tiny, fixed amount of memory. It trades a very small error margin for huge memory savings. Websites use it to count unique visitors, unique searches, or unique device IDs across millions of events.

The Stadium Gate Analogy

Suppose a stadium wants to know how many different people entered today. Writing every name on paper takes thousands of pages. A smarter method records only a few clues about each arrival and uses math to estimate the total. HyperLogLog follows this second method. It never stores the names. It keeps a compact summary and gives an estimate.

  Exact counting (Set)              HyperLogLog
  ┌────────────────────────┐        ┌────────────────────────┐
  │ Stores every visitor   │        │ Stores a small summary │
  │ 1 million IDs          │        │ Always about 12 KB     │
  │ Memory: tens of MB     │        │ Error: about 0.81%     │
  └────────────────────────┘        └────────────────────────┘

PFADD – Adding Items

The PF prefix honors Philippe Flajolet, who invented the algorithm. PFADD adds one or more items to the structure.

127.0.0.1:6379> PFADD visitors:today "user:1" "user:2" "user:3"
(integer) 1
127.0.0.1:6379> PFADD visitors:today "user:2" "user:4"
(integer) 1

PFADD returns 1 when the add changed the summary and 0 when it did not. The repeated "user:2" counts only once.

PFCOUNT – Reading the Estimate

127.0.0.1:6379> PFCOUNT visitors:today
(integer) 4

Small counts usually come back exact. Large counts carry a standard error of about 0.81 percent. A true total of 1,000,000 may show as 1,008,000 or 992,000.

PFMERGE – Combining Counts

PFMERGE joins several HyperLogLogs into one. Overlapping items count once.

127.0.0.1:6379> PFADD visitors:mon "a" "b" "c"
(integer) 1
127.0.0.1:6379> PFADD visitors:tue "b" "c" "d"
(integer) 1
127.0.0.1:6379> PFMERGE visitors:week visitors:mon visitors:tue
OK
127.0.0.1:6379> PFCOUNT visitors:week
(integer) 4

Monday:    a  b  c
Tuesday:      b  c  d
              ────────
Merged:    a  b  c  d   → 4 unique visitors

Memory Comparison

Method            1,000 items    1 million items    100 million items
────────────────  ─────────────  ─────────────────  ─────────────────
Set               ~ 60 KB        ~ 60 MB            ~ 6 GB
HyperLogLog       12 KB max      12 KB max          12 KB max

A HyperLogLog never grows past about 12 KB, no matter how many items you add.

Practical Example – Unique Visitors per Article

Each article gets its own HyperLogLog key.

Reader opens article 501:
  PFADD article:501:readers "visitor:88231"

Page shows "Readers so far":
  PFCOUNT article:501:readers

Editor wants total readers across three articles:
  PFCOUNT article:501:readers article:502:readers article:503:readers

PFCOUNT accepts several keys and returns the unique total across all of them.

When to Choose HyperLogLog

Good Fit

• Unique visitor counts on busy sites.
• Unique search terms per day.
• Reports where a 1 percent error causes no harm.

Poor Fit

• Billing, payments, or any count that must match exactly.
• Cases where you must list the members or check whether one item exists.

HyperLogLog cannot return stored items. Use a Set when you need the actual members.

Key Points

• HyperLogLog estimates unique counts with a fixed memory cost near 12 KB.
• PFADD inserts items, PFCOUNT reads the estimate, and PFMERGE combines structures.
• The standard error stays near 0.81 percent.
• The structure stores no items, so it cannot list or look them up.