Vector Filtering and Metadata

Pure vector similarity search is powerful, but real applications often need additional constraints. A shopper searching for "running shoes" usually also wants to filter by price range, brand, or size. This topic covers metadata storage and filtered search.

What Metadata Is

Metadata is structured information you attach to each vector alongside its embedding. Think of it as a small set of regular database fields stored next to the vector — searchable with exact-match logic, unlike the vector itself.

A SINGLE VECTOR RECORD:
─────────────────────────────────────────────────
{
  "id": "shoe-4521",
  "values": [0.23, 0.87, 0.45, ...],     ← the vector (1536 numbers)
  "metadata": {                           ← structured fields
    "name": "Trail Runner Pro",
    "brand": "Nike",
    "price": 89.99,
    "category": "running",
    "in_stock": true,
    "release_year": 2025
  }
}

Pre-Filtering vs. Post-Filtering

Vector databases apply filters in one of two ways, and the difference matters for both speed and correctness.

Post-Filtering (Naive Approach)

STEP 1: Run vector search → get top 10 results by similarity
STEP 2: Remove results that don't match the filter
STEP 3: Return whatever survives

PROBLEM:
  If 8 of the top 10 don't match the filter,
  you're left with only 2 results — even though
  better matches exist further down the ranking.

Pre-Filtering (Correct Approach)

STEP 1: Apply the filter FIRST → narrow the candidate pool
STEP 2: Run vector search ONLY within that filtered pool
STEP 3: Return top-k from the filtered, ranked results

RESULT: Always returns the best k matches that satisfy the filter

Modern vector databases (Pinecone, Weaviate, Qdrant, Milvus) all use pre-filtering by default. This is one of the key engineering challenges these platforms solve — efficiently combining metadata filters with approximate nearest neighbor search without losing speed.

Common Filter Operators

Operator	Meaning	Example
$eq	Equals	category = "running"
$ne	Not equal	brand != "Nike"
$gt / $gte	Greater than / or equal	price > 50
$lt / $lte	Less than / or equal	price < 100
$in	Matches any value in a list	brand in ["Nike", "Adidas"]
$and / $or	Combine multiple conditions	price < 100 AND in_stock = true

Filtered Search Example (Pinecone)

results = index.query(
    vector=query_vector,
    top_k=10,
    filter={
        "$and": [
            {"category": {"$eq": "running"}},
            {"price": {"$lte": 100}},
            {"in_stock": {"$eq": True}}
        ]
    },
    include_metadata=True
)

This query finds the closest vectors to your search query, restricted only to in-stock running shoes priced at $100 or below.

A Practical E-Commerce Example

USER SEARCH: "comfortable shoes for long walks"
USER FILTERS: Price under $80, Size 10, In stock only

QUERY FLOW:
─────────────────────────────────────────────────
1. Embed: "comfortable shoes for long walks"
       → query_vector

2. Apply filters BEFORE ranking:
       price <= 80 AND size = 10 AND in_stock = true
       → narrows 50,000 products to 1,200 candidates

3. Run vector similarity search WITHIN those 1,200
       → ranks by closeness to query_vector

4. Return top 10 most relevant matches
       → all guaranteed to satisfy price/size/stock filters

Multi-Tenancy Using Metadata

Metadata filtering also solves multi-tenancy — keeping different customers' data separate within one shared index. Add a tenant ID field and filter by it on every query.

# Storing data for tenant "company-A"
index.upsert(vectors=[{
    "id": "doc-1",
    "values": embedding,
    "metadata": {"tenant_id": "company-A", "text": "..."}
}])

# Querying only company-A's data
results = index.query(
    vector=query_vector,
    filter={"tenant_id": {"$eq": "company-A"}},
    top_k=5
)

This pattern is common in SaaS applications where many customers share the same underlying vector database, but each must only ever see their own data.

Choosing What to Store as Metadata

Store as Metadata	Embed Into the Vector
Price, category, date, status flags	Product description, article text
User ID, tenant ID	User reviews, comments
Exact values you filter or sort by	Anything you want to search by meaning

A simple rule guides this decision: if you need exact matching or range comparisons, use metadata. If you need fuzzy, meaning-based matching, embed it into the vector.

Metadata Size Limits

Most vector databases cap metadata size per record — commonly around 40KB. Store identifiers and short fields as metadata, and keep large text content in a separate database or file storage, referencing it by ID in the metadata instead of duplicating the full text.

Previous lesson

Back to course

Next lesson