Deep Learning Word Embeddings

Neural networks work with numbers, and words are not numbers. A word embedding turns each word into a list of numbers that captures its meaning. Words with similar meanings receive similar lists, which lets a network recognize that happy and joyful belong together even if it never saw them side by side.

Why Simple Numbering Fails

The simplest idea assigns each word an ID, such as cat = 1, dog = 2, car = 3. IDs carry no meaning. The network would treat cat and dog as no closer than cat and car. A second idea, one-hot encoding, uses a long list of zeros with a single one. It wastes space and still says nothing about meaning.

One-hot with a 5-word vocabulary:
  cat  = [1, 0, 0, 0, 0]
  dog  = [0, 1, 0, 0, 0]
  car  = [0, 0, 1, 0, 0]

Real vocabularies hold 50,000+ words, so each list becomes huge
and every word looks equally different from every other word.

The Embedding Idea

An embedding gives each word a short list of numbers, often 100 to 300 values. Each position in the list acts like a hidden trait. The network learns the traits by itself during training.

Imagine 3 traits: [is animal, is vehicle, is friendly]

cat   = [0.9, 0.0, 0.7]
dog   = [0.9, 0.0, 0.9]
car   = [0.0, 0.9, 0.1]

cat and dog sit close together. car sits far away.

Real embeddings use hundreds of traits that humans cannot label, but the principle stays the same.

Words as Points on a Map

Picture a map where every word is a dot. Related words cluster into neighbourhoods.

          Animals                     Vehicles
        cat   dog                   car   truck
          lion                        bus

     Fruits                   Feelings
   apple  mango             happy   joyful
     banana                   sad

How Embeddings Are Learned

A famous principle guides the training: words that appear in similar contexts have similar meanings. The words coffee and tea both appear near cup, drink, and hot, so they end up close on the map.

Word2Vec

Word2Vec trains a small network to predict a word from its neighbours, or predict neighbours from a word. The network weights that result become the embeddings.

Sentence: "I drink hot coffee every morning"

Training question: given "coffee", predict nearby words
  -> drink, hot, every, morning

GloVe

GloVe builds embeddings from counts of how often words appear together across a huge text collection.

Embedding Layers

Most modern networks include an embedding layer that learns its values together with the rest of the model for the task at hand.

Math with Meaning

Good embeddings capture relationships as directions on the map. The classic example subtracts and adds word lists.

king - man + woman = queen

The direction from "man" to "woman" matches
the direction from "king" to "queen"

Measuring Similarity

Cosine similarity compares the direction of two lists. A score near 1 means very similar, 0 means unrelated, and negative values mean opposite.

similarity(cat, dog)   = 0.80
similarity(cat, car)   = 0.15

Using an Embedding Layer

model = keras.Sequential([
    layers.Embedding(input_dim=10000, output_dim=64),
    layers.GlobalAveragePooling1D(),
    layers.Dense(1, activation="sigmoid")
])

This model maps up to 10,000 words to 64-number lists, averages them for each sentence, and predicts a yes-or-no label.

From Static to Contextual Embeddings

Classic embeddings give one list per word. The word bank gets the same list in river bank and money bank. Transformers create contextual embeddings, which change with the surrounding sentence, so each use of bank receives its own meaning.

Key Terms

• Embedding - a list of numbers that represents a word
• Vocabulary - the set of words a model knows
• Word2Vec and GloVe - popular methods for learning embeddings
• Cosine similarity - a score that compares two embeddings
• Contextual embedding - an embedding that changes with sentence context