TensorFlow RNN Basics
A Recurrent Neural Network (RNN) processes sequential data — text, audio, time series, video frames — where the order of elements matters. An RNN processes one element at a time and carries a memory of what it has seen so far (called the hidden state) into the processing of the next element. This memory is what makes RNNs fundamentally different from Dense and CNN layers, which treat each input independently.
Why Order Matters: The Story Analogy
The sentence "The dog bit the man" and "The man bit the dog" contain exactly the same words but have completely different meanings. A Dense layer treating each word independently cannot distinguish between them. An RNN reads words left to right, maintaining a growing memory of context with each step. By the time it processes "man," it already knows whether "dog" appeared before or after "bit."
The RNN Mechanism
At each time step t, the RNN computes: hidden_state(t) = tanh( W_h × hidden_state(t-1) + W_x × input(t) + b ) Where: W_h = weight matrix for the previous hidden state W_x = weight matrix for the current input b = bias hidden_state = a vector that accumulates sequence memory tanh = squashes output to (-1, 1) The same weights W_h and W_x are used at EVERY time step. This weight sharing is how RNNs learn patterns at any position.
Unrolling an RNN Through Time
Sequence: ["The", "dog", "bit", "the", "man"] h_0 = zeros (initial hidden state) Step 1: input="The" + h_0 → [W_x × "The" + W_h × h_0] → h_1 Step 2: input="dog" + h_1 → [W_x × "dog" + W_h × h_1] → h_2 Step 3: input="bit" + h_2 → [W_x × "bit" + W_h × h_2] → h_3 Step 4: input="the" + h_3 → [W_x × "the" + W_h × h_3] → h_4 Step 5: input="man" + h_4 → [W_x × "man" + W_h × h_4] → h_5 h_5 is the final hidden state — a summary of the entire sequence. Use h_5 for sequence classification. Use each h_t for sequence-to-sequence tasks.
SimpleRNN in TensorFlow
import tensorflow as tf
import numpy as np
# Create a simple RNN model for sequence classification
model = tf.keras.Sequential([
# Embedding layer converts integer token IDs to dense vectors
tf.keras.layers.Embedding(input_dim=10000, output_dim=64, input_length=100),
# SimpleRNN processes the sequence
tf.keras.layers.SimpleRNN(128),
# Output layer
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()
return_sequences Parameter
# return_sequences=False (default) # Returns only the final hidden state # Shape: (batch, units) # Use for: classification, regression on a whole sequence tf.keras.layers.SimpleRNN(64, return_sequences=False) # input shape: (batch, 100, embedding_dim) # output shape: (batch, 64) # return_sequences=True # Returns hidden state at every time step # Shape: (batch, timesteps, units) # Use for: stacking RNN layers, sequence labeling, seq-to-seq tf.keras.layers.SimpleRNN(64, return_sequences=True) # input shape: (batch, 100, embedding_dim) # output shape: (batch, 100, 64)
Stacking RNN Layers
model = tf.keras.Sequential([
tf.keras.layers.Embedding(10000, 64, input_length=100),
# First RNN — must return sequences for next layer to consume
tf.keras.layers.SimpleRNN(128, return_sequences=True),
tf.keras.layers.Dropout(0.2),
# Second RNN — returns only final state
tf.keras.layers.SimpleRNN(64),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1, activation='sigmoid')
])
Bidirectional RNN
A standard RNN reads the sequence left to right. A Bidirectional RNN runs two RNNs: one left-to-right and one right-to-left. Their hidden states are concatenated at each step. The model learns context from both directions simultaneously — knowing what comes after a word is just as valuable as knowing what came before.
# Bidirectional wrapper doubles the output size
tf.keras.layers.Bidirectional(tf.keras.layers.SimpleRNN(64))
# output shape: (batch, 128) ← 64 forward + 64 backward
model = tf.keras.Sequential([
tf.keras.layers.Embedding(10000, 64, input_length=100),
tf.keras.layers.Bidirectional(tf.keras.layers.SimpleRNN(64)),
tf.keras.layers.Dense(1, activation='sigmoid')
])
The Vanishing Gradient Problem in RNNs
Long sequence processing in SimpleRNN:
"I grew up in France ... I speak fluent ___"
↑ ↑
position 1 position 47
At position 47, the gradient signal for the word "France"
(position 1) must travel back through 46 tanh operations.
Each tanh operation squashes gradients by up to 0.42×.
After 46 steps: gradient ≈ 0.42^46 ≈ 10^-17 (essentially zero)
The RNN effectively forgets long-range context.
This is why SimpleRNN is rarely used in production. LSTM and GRU layers (covered next) solve the vanishing gradient problem with gating mechanisms that control how much information is kept or discarded at each step.
When to Use RNNs
Task Architecture ─────────────────────────────────────────────────────────── Sentiment classification Embedding → LSTM → Dense Text generation Embedding → LSTM(return_seq) → Dense Time series forecasting LSTM or GRU Named entity recognition Bidirectional LSTM (return_seq) → Dense Machine translation Encoder-Decoder LSTM Speech recognition Bidirectional LSTM or Transformer ────────────────────────────────────────────────────────────
Input Shape for RNNs
RNN expects 3D input: (batch_size, timesteps, features)
Text example:
batch_size = 32 sentences per batch
timesteps = 100 words per sentence
features = 64 embedding dimensions per word
Shape: (32, 100, 64)
Time series example:
batch_size = 64 windows per batch
timesteps = 30 days per window
features = 5 sensor readings per day
Shape: (64, 30, 5)
Feeding numpy array directly:
x = np.random.random((1000, 30, 5)).astype('float32')
y = np.random.randint(0, 2, size=1000)
model.fit(x, y, epochs=10, batch_size=64)
SimpleRNN introduces the core RNN concept — a hidden state that carries memory across time steps — but vanishes on long sequences. The next topic covers LSTM, which extends SimpleRNN with cell states and gates that allow the network to remember information across hundreds or thousands of steps without gradient degradation.