Vector Embeddings Deep Dive

Embeddings are the foundation of every vector database application. This topic goes beyond the basics and shows you how embeddings encode meaning, what makes one embedding model better than another, and how to pick the right model for your project.

What an Embedding Actually Encodes

Each number in an embedding vector represents a learned feature — a dimension of meaning the model discovered during training. No one programs these features manually. The model finds them automatically by studying patterns across billions of examples.

Word: "Paris"

Embedding (simplified to 6 dimensions):
┌─────────────────┬────────────────────┬───────┐
│ Dimension       │ What it captures   │ Value │
├─────────────────┼────────────────────┼───────┤
│ Dim 1           │ Is it a place?     │  0.95 │
│ Dim 2           │ Is it in Europe?   │  0.91 │
│ Dim 3           │ Is it a capital?   │  0.88 │
│ Dim 4           │ Is it cultural?    │  0.85 │
│ Dim 5           │ Is it a person?    │  0.02 │
│ Dim 6           │ Is it abstract?    │  0.04 │
└─────────────────┴────────────────────┴───────┘
(Real embeddings have 768–1536 dims — each capturing subtler features)

The Famous King – Queen Example

One of the most celebrated properties of embeddings is that you can do arithmetic on them and get meaningful results.

  "King" − "Man" + "Woman" ≈ "Queen"

  VECTOR SPACE VIEW:
  ─────────────────────────────────────────
       ● Queen         ● King
            ↑               ↑
            │ +Woman        │ +Woman
            │               │
       ● Girl          ● Boy
  ─────────────────────────────────────────

  The "gender direction" and "royalty direction" are
  separate, learnable axes in the embedding space.

This arithmetic works because embeddings capture relationships, not just identities. Words with similar relationships end up with similar vector offsets.

Types of Embedding Models

Word Embeddings

Older models like Word2Vec and GloVe create one vector per word. The word "bank" gets a single vector, regardless of whether you mean a river bank or a financial bank. This ambiguity limits their usefulness in modern applications.

Contextual Embeddings

Models like BERT create a different vector for the same word depending on its surrounding context. "Bank" in "I went to the bank to deposit money" produces a different vector than "bank" in "We sat by the river bank." This context-awareness makes them far more powerful.

Sentence and Document Embeddings

Models like Sentence-BERT and OpenAI's text-embedding models encode entire sentences or paragraphs into a single vector. These are the models you use most often with vector databases.

Cross-Modal Embeddings

Some models embed different data types into the same vector space. CLIP, developed by OpenAI, places images and text in a shared space. This means you can search images using text — and it works remarkably well.

CLIP SHARED VECTOR SPACE:
──────────────────────────────────────────────
  Text: "a dog playing in snow"   → [0.3, 0.9, 0.1, ...]
  Photo of dog in snow            → [0.3, 0.9, 0.1, ...]
                                         ↑ Nearly identical!

  This is why Google Photos can find your
  winter dog photos when you type "dog in snow."

Choosing an Embedding Model

The Consistency Rule

Every vector in your database must come from the same embedding model. Mixing models corrupts your search results because different models use completely different vector spaces. A cosine score between two vectors from different models is meaningless.

  WRONG ✗                           CORRECT ✓
  ──────────────────────────────    ──────────────────────────────
  Doc A → Model X → vector          Doc A → Model X → vector
  Doc B → Model Y → vector          Doc B → Model X → vector
  Query → Model X → vector          Query → Model X → vector
       ↑                                  ↑
  Mixed spaces = garbage results    Same space = valid results

When you switch embedding models, re-embed your entire dataset from scratch. There is no shortcut.