TensorFlow Text Sequences

Neural networks operate on numbers, not words. Converting raw text into numerical sequences that RNNs and Transformers can process requires a pipeline of tokenization, vocabulary building, sequence padding, and embedding. This topic covers the complete text preprocessing workflow in TensorFlow, from a raw string like "The cat sat on the mat" to a tensor ready for model input.

The Translation Analogy

Text preprocessing is like translating a book into a coded language before sending it through a processing machine. Each word gets a unique ID number. The machine works with these ID numbers. After processing, the output IDs can be translated back to words if needed. The quality of the translation (vocabulary coverage, token granularity) directly affects how well the machine understands the book.

Step 1 — Tokenization: Words to Integer IDs

import tensorflow as tf

# TextVectorization layer handles tokenization automatically
vectorize_layer = tf.keras.layers.TextVectorization(
    max_tokens=10000,          # Keep only the 10,000 most common words
    output_mode='int',         # Output integer IDs (for Embedding layer)
    output_sequence_length=200 # Pad/truncate all sequences to this length
)

# Build the vocabulary from training text
train_texts = [
    "I love this movie, it was fantastic!",
    "Terrible film. Waste of time.",
    "Amazing acting and great story.",
    "The worst movie I have ever seen."
]

vectorize_layer.adapt(train_texts)

# Check vocabulary size
print(len(vectorize_layer.get_vocabulary()))  # Up to 10,000
print(vectorize_layer.get_vocabulary()[:10])  # Most common words first
# ['', '[UNK]', 'the', 'i', 'movie', 'was', 'this', ...]

Step 2 — Convert Text to Integer Sequences

# Convert one sentence
sample = tf.constant(["I love this movie"])
sequence = vectorize_layer(sample)
print(sequence)
# tf.Tensor([[4  3  7  5  0  0  0 ... ]], shape=(1, 200), dtype=int64)
# Words map to IDs; zeros are padding

# Decode back: ID 4="i", 3="love", 7="this", 5="movie"
vocab = vectorize_layer.get_vocabulary()
decoded = [vocab[id] for id in sequence[0].numpy() if id > 0]
print(decoded)  # ['i', 'love', 'this', 'movie']

Special Tokens

Index 0 → '' (padding token — fills positions to reach sequence length)
Index 1 → '[UNK]' (unknown — words not in the vocabulary)
Index 2+ → actual vocabulary words, sorted by frequency

Step 3 — Embedding: IDs to Dense Vectors

An integer ID like 42 carries no meaning by itself — ID 42 and ID 43 are not necessarily related. An Embedding layer maps each ID to a dense vector of learned floating-point numbers. Words with similar meanings end up with similar vectors after training.

# Embedding layer: maps each integer ID to a vector
embedding = tf.keras.layers.Embedding(
    input_dim=10000,    # Vocabulary size (max_tokens from TextVectorization)
    output_dim=128,     # Vector dimension per word
    input_length=200    # Sequence length (output_sequence_length from above)
)

Diagram — Embedding transformation:

Input:   [4,  3,  7,   5,  0,  0, ...]   (integer IDs)
           ↓   ↓   ↓    ↓   ↓   ↓
Output:  [v4, v3, v7, v5, v0, v0, ...]   (128-dim vectors each)

v4 = [0.23, -0.41, 0.87, ..., 0.12]   # vector for "i"
v3 = [0.91, 0.34, -0.22, ..., 0.67]   # vector for "love"
...

Shape change: (batch, 200) → (batch, 200, 128)

Padding: Making Sequences the Same Length

texts = [
    "short text",                          # 2 words
    "this is a medium length sentence",    # 6 words
    "this is quite a long sentence with many more words added to it"  # 13 words
]

# TextVectorization with output_sequence_length=10:
# [short, text, 0,    0,    0,    0,    0,    0,    0,    0   ]
# [this,  is,   a,    medium,length,sentence,0,    0,    0,    0   ]
# [this,  is,   quite,a,    long, sentence,with,many, more, words  ] ← truncated

# padding='post' — zeros added at end (default for TextVectorization)
# padding='pre'  — zeros added at start (sometimes better for classification)

Manual Padding with pad_sequences

sequences = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]

padded = tf.keras.preprocessing.sequence.pad_sequences(
    sequences,
    maxlen=5,
    padding='post',    # Pad at end: [1,2,3,0,0]
    truncating='post', # Truncate at end if too long
    value=0            # Padding value
)

print(padded)
# [[1, 2, 3, 0, 0],
#  [4, 5, 0, 0, 0],
#  [6, 7, 8, 9, 0]]  ← length 4, padded to 5

Building a Full Text Pipeline

import tensorflow as tf

MAX_TOKENS = 10000
SEQUENCE_LENGTH = 200
EMBEDDING_DIM = 128

# Vectorization layer
vectorize_layer = tf.keras.layers.TextVectorization(
    max_tokens=MAX_TOKENS,
    output_mode='int',
    output_sequence_length=SEQUENCE_LENGTH
)

# Adapt on training texts
vectorize_layer.adapt(train_texts)

# Build model with vectorization as first layer
model = tf.keras.Sequential([
    tf.keras.Input(shape=(1,), dtype=tf.string),
    vectorize_layer,
    tf.keras.layers.Embedding(MAX_TOKENS, EMBEDDING_DIM),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train directly on raw text strings — no preprocessing needed!
model.fit(train_texts, train_labels, epochs=10)

# Predict on raw text
model.predict(["This film was absolutely terrible"])

Subword Tokenization

Word-level tokenization assigns one token per word. This means "running," "runner," and "runs" are treated as completely separate and unrelated tokens. Subword tokenization breaks words into common pieces, so "running" becomes ["run", "##ning"], sharing the "run" token with "runner" and "runs." This approach handles unseen words and reduces vocabulary size.

# TensorFlow Text provides subword tokenizers
# pip install tensorflow-text

import tensorflow_text as text

# Bert tokenizer (pre-built vocabulary)
tokenizer = text.BertTokenizer('vocab.txt')
tokens = tokenizer.tokenize(["running quickly"])
print(tokens)  # [[['run', '##ning'], ['quick', '##ly']]]

Pre-trained Word Embeddings

# Load GloVe embeddings (pre-trained on 6B words)
# Download: glove.6B.100d.txt from Stanford NLP

import numpy as np

embedding_index = {}
with open('glove.6B.100d.txt', encoding='utf-8') as f:
    for line in f:
        word, *vector = line.split()
        embedding_index[word] = np.array(vector, dtype='float32')

vocab = vectorize_layer.get_vocabulary()
embedding_matrix = np.zeros((len(vocab), 100))

for i, word in enumerate(vocab):
    vec = embedding_index.get(word)
    if vec is not None:
        embedding_matrix[i] = vec

# Initialize embedding layer with pre-trained weights
embedding_layer = tf.keras.layers.Embedding(
    input_dim=len(vocab),
    output_dim=100,
    weights=[embedding_matrix],
    trainable=False   # Freeze pre-trained embeddings
)

Text sequence preprocessing is the foundation of all NLP work in TensorFlow. The pipeline — tokenize → pad → embed — converts unstructured text into structured numerical tensors that any sequence model can learn from. The next topic applies this knowledge to sentiment analysis: a complete project classifying movie reviews as positive or negative.