PyTorch Sequence Modeling

Sequence modeling covers tasks where the order of data points matters and where inputs or outputs can vary in length. Language, audio, time series, and video are all sequences. PyTorch provides tools to handle these tasks — from padding variable-length sequences to building encoder-decoder architectures.

Types of Sequence Tasks

Task Type            Input → Output       Example
──────────────────────────────────────────────────────────────
Many-to-one          seq → single value   Sentiment analysis
One-to-many          single → seq         Image captioning
Many-to-many (same)  seq → seq (aligned)  Named entity tagging
Many-to-many (diff)  seq → seq (diff len) Machine translation

Handling Variable-Length Sequences

Real datasets contain sequences of different lengths. A batch of sentences might have 5, 23, 41, and 12 words respectively. Tensors require all samples in a batch to have the same size. The solution is padding — adding zeros to shorter sequences to match the longest in the batch.

from torch.nn.utils.rnn import pad_sequence
import torch

# Three sequences of different lengths
s1 = torch.tensor([1.0, 2.0, 3.0])
s2 = torch.tensor([4.0, 5.0])
s3 = torch.tensor([6.0, 7.0, 8.0, 9.0])

padded = pad_sequence([s1, s2, s3], batch_first=True, padding_value=0.0)
print(padded)
# tensor([[1., 2., 3., 0.],
#         [4., 5., 0., 0.],
#         [6., 7., 8., 9.]])

PackedSequence — Efficient Variable-Length Processing

Padding wastes computation on zero values. PyTorch's PackedSequence packs sequences together and tells the RNN to skip the padding. This saves time on long padded sequences.

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
import torch
import torch.nn as nn

gru = nn.GRU(input_size=4, hidden_size=8, batch_first=True)

# Sequences with actual lengths [4, 2, 3]
x       = torch.rand(3, 4, 4)       # batch=3, max_len=4, features=4
lengths = torch.tensor([4, 2, 3])   # true length of each sequence

# Pack before passing to RNN
packed = pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False)
output_packed, h_n = gru(packed)

# Unpack to get output tensor
output, out_lengths = pad_packed_sequence(output_packed, batch_first=True)
print(output.shape)   # torch.Size([3, 4, 8])

Many-to-One: Sentence Classifier

class SentimentModel(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_size):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.gru   = nn.GRU(embed_dim, hidden_size, batch_first=True)
        self.fc    = nn.Linear(hidden_size, 2)  # positive / negative

    def forward(self, x):
        emb = self.embed(x)           # [B, seq_len, embed_dim]
        _, h = self.gru(emb)          # h: [1, B, hidden]
        return self.fc(h.squeeze(0))  # [B, 2]

Many-to-Many: Token Labeling

For tasks like named entity recognition (NER), you need a label for every token in the input sequence. Use all outputs, not just the final hidden state.

class NERModel(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_size, num_tags):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm  = nn.LSTM(embed_dim, hidden_size,
                             batch_first=True, bidirectional=True)
        self.fc    = nn.Linear(hidden_size * 2, num_tags)

    def forward(self, x):
        emb     = self.embed(x)        # [B, seq_len, embed_dim]
        out, _  = self.lstm(emb)       # [B, seq_len, hidden*2]
        return self.fc(out)            # [B, seq_len, num_tags]

model = NERModel(vocab_size=5000, embed_dim=64, hidden_size=128, num_tags=9)
tokens = torch.randint(0, 5000, (8, 30))  # 8 sentences, 30 tokens each
tags   = model(tokens)
print(tags.shape)   # torch.Size([8, 30, 9])  ← one tag distribution per token

Sequence-to-Sequence Architecture

Machine translation and text summarization map one sequence to a different sequence. The standard approach uses an encoder that reads the input and compresses it into a context vector, then a decoder that generates the output sequence one token at a time.

Input:  "Hello world"  (English)
           │
     ┌─────┴─────┐
     │  Encoder  │ ← reads input sequence
     └─────┬─────┘
           │ context vector (final hidden state)
     ┌─────┴─────┐
     │  Decoder  │ ← generates output one token at a time
     └─────┬─────┘
           │
Output: "Hola mundo"  (Spanish)

Embedding Layer for Sequences

Text sequences contain integer token IDs. Before passing them to an RNN, embed them into dense vectors using nn.Embedding. The embedding layer learns a continuous vector representation for each word.

embed = nn.Embedding(num_embeddings=10000,  # vocabulary size
                     embedding_dim=128,     # vector size per word
                     padding_idx=0)         # token 0 = padding (zero vector)

tokens = torch.tensor([[1, 2, 3, 0, 0],
                        [4, 5, 0, 0, 0]])   # padded token sequences

vectors = embed(tokens)
print(vectors.shape)   # torch.Size([2, 5, 128])

Masking Padded Positions in Loss

When computing loss for a sequence-to-sequence model, you should not penalize predictions on padding tokens. Use ignore_index in the loss function to mask them.

loss_fn = nn.CrossEntropyLoss(ignore_index=0)   # 0 = padding token index

Summary

Sequence modeling handles tasks where input or output is an ordered series of items. Use padding and PackedSequence to handle batches of variable-length sequences. For classification of an entire sequence, use only the final RNN hidden state. For token-level tasks like NER, use all RNN output steps. Use nn.Embedding to convert token IDs to continuous vectors before feeding them to an RNN. Sequence-to-sequence tasks use an encoder-decoder architecture. Mask padding positions in the loss with ignore_index to avoid training on empty tokens.