PyTorch RNN Basics

A Recurrent Neural Network (RNN) processes sequences — one element at a time — while maintaining a hidden state that carries information from previous steps. CNNs process grid data; RNNs process ordered sequences: sentences word by word, stock prices day by day, sensor readings second by second.

Why Standard Networks Fail on Sequences

A fully connected network takes a fixed-size input and produces a fixed-size output. It has no notion of order or time. If you feed it the words of a sentence one at a time, each word is processed independently — the network forgets what came before. An RNN solves this by passing a hidden state from one step to the next.

Standard network:
  word1 → model → output1   (forgets word1)
  word2 → model → output2   (forgets word2)
  No memory between steps.

RNN:
  word1 → RNN → h1 (hidden state carries memory)
  word2 + h1 → RNN → h2
  word3 + h2 → RNN → h3
  h3 → output   (final hidden state summarizes whole sequence)

The RNN Cell — One Step

At each time step, an RNN cell receives two inputs: the current input x_t and the previous hidden state h_{t-1}. It combines them and produces a new hidden state h_t.

h_t = tanh(W_h × h_{t-1} + W_x × x_t + b)

where:
  x_t   = current input
  h_{t-1} = previous hidden state
  W_h   = weight matrix for hidden-to-hidden connection
  W_x   = weight matrix for input-to-hidden connection
  b     = bias
  tanh  = activation function

Diagram of one RNN step:

  h_{t-1} ──→  ┌─────────┐ ──→ h_t
               │  RNN    │
    x_t   ──→  │  Cell   │
               └─────────┘

Unrolled RNN Over a Sequence

Sequence: ["I", "love", "PyTorch"]

Step 1:  x="I"       + h_0 (zeros) → h_1
Step 2:  x="love"    + h_1         → h_2
Step 3:  x="PyTorch" + h_2         → h_3 (final hidden state)

h_3 contains a compressed representation of the entire sentence.

PyTorch nn.RNN

import torch
import torch.nn as nn

rnn = nn.RNN(
    input_size=10,    # size of each input vector (e.g. word embedding size)
    hidden_size=20,   # size of hidden state
    num_layers=1,     # number of stacked RNN layers
    batch_first=True  # input shape: [batch, seq_len, features]
)

# Input: batch=4, sequence_length=7, features=10
x = torch.rand(4, 7, 10)
h0 = torch.zeros(1, 4, 20)   # initial hidden state: [num_layers, batch, hidden_size]

output, h_n = rnn(x, h0)

print(output.shape)   # torch.Size([4, 7, 20])  ← hidden state at every step
print(h_n.shape)      # torch.Size([1, 4, 20])  ← final hidden state

Stacked RNNs

Setting num_layers > 1 stacks multiple RNN layers. The hidden state output of layer 1 becomes the input to layer 2. Deeper RNNs can learn more complex sequential patterns.

rnn = nn.RNN(input_size=10, hidden_size=20, num_layers=2, batch_first=True)

x  = torch.rand(4, 7, 10)
h0 = torch.zeros(2, 4, 20)    # 2 layers

output, h_n = rnn(x, h0)
print(h_n.shape)   # torch.Size([2, 4, 20])  ← one hidden state per layer

Bidirectional RNN

A bidirectional RNN processes the sequence in both directions — left to right and right to left. The hidden states from both directions are concatenated. This gives each step access to both past and future context.

rnn = nn.RNN(input_size=10, hidden_size=20, batch_first=True, bidirectional=True)

x = torch.rand(4, 7, 10)
output, h_n = rnn(x)

print(output.shape)   # torch.Size([4, 7, 40])  ← 20 forward + 20 backward
print(h_n.shape)      # torch.Size([2, 4, 20])  ← 2 directions

Using the RNN for Classification

class RNNClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc  = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        output, h_n = self.rnn(x)
        # Use the final hidden state for classification
        last_hidden = h_n.squeeze(0)     # [batch, hidden_size]
        return self.fc(last_hidden)

model = RNNClassifier(input_size=10, hidden_size=32, num_classes=5)
x = torch.rand(8, 15, 10)    # 8 sequences, each 15 steps long, 10 features
out = model(x)
print(out.shape)   # torch.Size([8, 5])

Vanishing Gradient Problem in RNNs

Long sequences cause gradients to vanish as they travel through many time steps during backpropagation through time. An RNN trained on a 100-step sequence must propagate gradients through 100 multiplications — the gradient for the first step is often essentially zero. This prevents the RNN from learning long-range dependencies. LSTM and GRU (covered in the next two topics) were designed specifically to fix this problem.

Summary

RNNs process sequences by maintaining a hidden state that passes information from one time step to the next. The RNN cell takes the current input and the previous hidden state, producing a new hidden state. Use nn.RNN with batch_first=True so input shape follows [batch, seq_len, features]. Stack RNN layers with num_layers. Use bidirectional=True for tasks where both past and future context matter. The final hidden state h_n summarizes the entire sequence and feeds into a classification or regression head. For long sequences, use LSTM or GRU to avoid the vanishing gradient problem.