Vector Scaling and Performance

A vector database that works well with 10,000 vectors can struggle badly at 100 million vectors without proper planning. This topic covers the techniques that keep vector search fast and affordable as your data grows.

Where Performance Problems Come From

THE THREE COST DRIVERS OF VECTOR SEARCH:
─────────────────────────────────────────────────
1. NUMBER OF VECTORS     → more data to search through
2. VECTOR DIMENSIONS     → more numbers per comparison
3. QUERY VOLUME          → more searches per second

Memory needed ≈ (vectors × dimensions × 4 bytes) + index overhead

Example:
  10 million vectors × 1536 dimensions × 4 bytes = 61.4 GB
  (before adding index structures, which add 20-50% more)

Sharding: Splitting Data Across Machines

Sharding divides your vector collection across multiple servers. Each shard holds a portion of the data, and queries run in parallel across shards.

WITHOUT SHARDING:                  WITH SHARDING (3 shards):
┌─────────────────────┐            ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Single Server       │            │ Shard 1  │ │ Shard 2  │ │ Shard 3  │
│ 100M vectors        │            │ 33M      │ │ 33M      │ │ 33M      │
│ Slow, memory-limited│            │ vectors  │ │ vectors  │ │ vectors  │
└─────────────────────┘            └──────────┘ └──────────┘ └──────────┘
                                          ↑              ↑              ↑
                                          └──────── Query runs in parallel ────────┘
                                               Results merged and re-ranked

Sharding strategies vary: some systems shard randomly (simple, even distribution), others shard by a logical key like tenant ID (keeps related data together, simplifies multi-tenancy).

Replication: Copies for Speed and Reliability

Replication creates multiple copies of the same data across servers. This serves two purposes: handling more queries per second, and protecting against server failure.

REPLICATION SETUP:
─────────────────────────────────────
   Incoming Queries
        │
        ▼
   Load Balancer
    ╱    │    ╲
   ╱     │     ╲
Replica 1  Replica 2  Replica 3
(same data, each handles 1/3 of traffic)

If Replica 2 fails → traffic reroutes to 1 and 3
No data loss, minimal disruption

Compression Techniques

Compression reduces memory footprint, often at a small accuracy cost. Covered in more depth in Topic 8, here is a summary of the main techniques.

Caching Frequent Queries

Many applications see repeated or similar queries. Caching avoids recomputing the same search repeatedly.

import hashlib
import json

query_cache = {}

def cached_search(query_text, top_k=5):
    cache_key = hashlib.md5(f"{query_text}-{top_k}".encode()).hexdigest()

    if cache_key in query_cache:
        return query_cache[cache_key]   # instant return, no DB hit

    query_vector = get_embedding(query_text)
    results = index.query(vector=query_vector, top_k=top_k)

    query_cache[cache_key] = results
    return results

In production, use a dedicated cache service like Redis instead of an in-memory Python dictionary, since the dictionary resets whenever your application restarts.

Batching Insert Operations

Inserting vectors one at a time wastes network round trips. Batch inserts dramatically improve indexing throughput.

SLOW: One vector per request
─────────────────────────────────
for vector in vectors:
    index.upsert(vectors=[vector])    # 10,000 separate API calls

FAST: Batched requests
─────────────────────────────────
batch_size = 100
for i in range(0, len(vectors), batch_size):
    batch = vectors[i:i + batch_size]
    index.upsert(vectors=batch)        # 100 API calls instead of 10,000

Monitoring the Metrics That Matter

A Scaling Roadmap

DATASET SIZE              RECOMMENDED APPROACH
─────────────────────────────────────────────────────────
Under 1M vectors           Single server, HNSW, no sharding needed
1M – 10M vectors           Single server with replication for traffic
10M – 100M vectors         Sharding + replication + quantization
100M+ vectors              Distributed cluster + PQ compression +
                            careful nlist/nprobe or M/ef tuning

Cost Optimization Tips

• Use lower-dimensional embedding models when your accuracy needs allow it (512 dims instead of 1536 cuts memory by 3x)
• Delete vectors you no longer need instead of leaving stale data in the index
• Use serverless or auto-scaling vector database tiers for unpredictable traffic patterns
• Apply metadata filters aggressively to shrink the search space before ranking

Scaling a vector database is rarely about one single fix. It comes from combining sharding, replication, compression, caching, and careful index tuning, each addressing a different part of the cost and performance equation.