Databricks Vector Search

Vector Search finds information by meaning instead of exact words. A shopper who types "warm jacket for snow" receives results about winter coats, even when the product pages never contain the word "warm." The technology powers smart search, recommendations, and the retrieval step of modern AI assistants.

The Library by Topic Analogy

Imagine a library that arranges books by subject instead of alphabet. Books about cooking stand together, and books about space sit in another area. A visitor who asks about "baking bread" walks to the cooking shelves and finds nearby titles on pastry and flour. Vector Search arranges information in a similar map of meaning.

A MAP OF MEANING (SIMPLIFIED)
──────────────────────────────────────────────────────────
                    ● winter coat
              ● snow jacket     ● warm parka
                                           (cluster: cold weather)

● running shoes     ● trail sneakers
      (cluster: footwear)

               ● sunscreen      ● beach towel
                    (cluster: summer)

Question: "warm jacket for snow" lands near the cold-weather cluster.

Embeddings: Numbers That Capture Meaning

An embedding is a list of numbers that represents the meaning of a piece of text. An embedding model reads the text and produces the list. Similar meanings produce similar lists. The search engine measures the distance between lists to find the closest matches.

TEXT TO NUMBERS
──────────────────────────────────────────────────────────
"winter coat"   ──► [0.82, 0.10, 0.45, ...]
"snow jacket"   ──► [0.80, 0.12, 0.47, ...]   (very close)
"beach towel"   ──► [0.05, 0.91, 0.20, ...]   (far away)

Parts of Databricks Vector Search

Vector Search Endpoint

The endpoint is the compute that stores and queries indexes. One endpoint can host several indexes.

Vector Index

The index stores the embeddings and supports fast similarity queries. Databricks offers two main types.

Index Type	How It Works
Delta Sync Index	Reads a Delta table and updates itself when the table changes
Direct Vector Access Index	Accepts vectors that your code inserts and updates directly

Embedding Model

A Delta Sync index can compute embeddings for you through a model endpoint. You can instead provide embeddings that you calculated yourself.

Prepare the Source Table

A Delta Sync index needs Change Data Feed on the source table, because the index reads row changes. The table also needs a unique key column and a text column.

CREATE TABLE main.shop.product_docs (
  product_id  BIGINT,
  title       STRING,
  description STRING
)
TBLPROPERTIES (delta.enableChangeDataFeed = true);

Create an Endpoint and Index

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()

vsc.create_endpoint(name="shop_vs_endpoint", endpoint_type="STANDARD")

index = vsc.create_delta_sync_index(
    endpoint_name="shop_vs_endpoint",
    index_name="main.shop.product_docs_index",
    source_table_name="main.shop.product_docs",
    pipeline_type="TRIGGERED",
    primary_key="product_id",
    embedding_source_column="description",
    embedding_model_endpoint_name="YOUR-EMBEDDING-ENDPOINT"
)

The triggered pipeline type updates the index when you request a sync. A continuous pipeline keeps the index current within seconds.

Search the Index

results = index.similarity_search(
    query_text="warm jacket for snow",
    columns=["product_id", "title"],
    num_results=3
)

print(results)

SEARCH FLOW
──────────────────────────────────────────────────────────
Query text ──► embedding model ──► query vector
                                       │
                         compare with stored vectors
                                       ▼
                         top 3 closest products returned

Filters and Hybrid Search

Filters narrow results by column values, such as price or category. Hybrid search combines meaning-based matching with keyword matching, which helps when users search for exact product codes or names.

Access Control

Unity Catalog permissions protect indexes. Grant users query access only on indexes they should use. Source tables follow their own rules, so sensitive text stays protected.

Example: Course Finder

An education site lists 8,000 short courses. Learners type goals such as "learn to analyze sales numbers" and receive courses on spreadsheets, SQL, and dashboards. The team indexes course descriptions and returns the top ten matches for each query.

COURSE FINDER
──────────────────────────────────────────────────────────
Courses table ──► Delta Sync Index
Learner goal  ──► Vector Search ──► ten closest courses ──► results page

Tips for Good Results

• Split long documents into short passages before indexing.
• Store titles and source links beside each passage.
• Test queries from real users and adjust the passage size.
• Re-sync the index after major table updates.

Key Points

• Vector Search finds items by meaning, not exact words.
• Embeddings convert text into numbers that capture meaning.
• A Delta Sync index follows a Delta table, and a Direct Vector Access index accepts vectors from code.
• Change Data Feed must be on for Delta Sync source tables.
• The similarity_search method returns the closest matches.
• Filters, hybrid search, and Unity Catalog permissions refine and protect results.