PyTorch Overfitting Prevention

Overfitting happens when a model memorizes training data instead of learning patterns that generalize. The model gets excellent scores on training data but performs poorly on data it has never seen. Recognizing and preventing overfitting is one of the most important practical skills in deep learning.

Identifying Overfitting

Healthy training:
  Epoch  Train Loss   Val Loss
    10     0.50        0.53     ← both decrease together
    20     0.31        0.34
    50     0.12        0.14

Overfitting:
  Epoch  Train Loss   Val Loss
    10     0.50        0.52     ← starts OK
    20     0.18        0.34     ← val loss starts climbing
    50     0.04        0.71     ← model memorized training data

Why Overfitting Happens

• Too few training samples relative to model size
• Model is too large (too many parameters for the task)
• Training too long without early stopping
• No regularization applied

Prevention Strategy 1: More Data

The most direct fix for overfitting is more training data. A model that memorizes 1,000 examples cannot memorize 100,000. When real data is scarce, data augmentation creates more variety from existing samples.

from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(15),
    transforms.ColorJitter(brightness=0.3, contrast=0.3),
    transforms.ToTensor(),
])

Prevention Strategy 2: Reduce Model Complexity

If your model has far more parameters than necessary for the task, it will find ways to memorize noise. Reduce the number of layers or the number of units per layer until validation loss stops worsening.

import torch.nn as nn

# Overly complex model for a simple task:
model_too_large = nn.Sequential(
    nn.Linear(10, 1024),
    nn.ReLU(),
    nn.Linear(1024, 512),
    nn.ReLU(),
    nn.Linear(512, 1)
)

# Appropriately sized:
model_right_size = nn.Sequential(
    nn.Linear(10, 32),
    nn.ReLU(),
    nn.Linear(32, 1)
)

Prevention Strategy 3: Dropout

Dropout randomly zeroes out a fraction of activations during training. This forces the network to learn redundant representations — no single neuron can be relied upon. The result is a more robust model that generalizes better. Dropout is covered in detail in the next topic.

model = nn.Sequential(
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.5),    # drop 50% of activations during training
    nn.Linear(128, 10)
)

Prevention Strategy 4: L2 Regularization (Weight Decay)

L2 regularization adds a penalty term to the loss that grows with the size of the model's weights. Large weights increase the penalty, so the optimizer keeps weights small. Small weights mean the model relies on many inputs rather than memorizing a few specific patterns.

import torch.optim as optim

optimizer = optim.Adam(
    model.parameters(),
    lr=0.001,
    weight_decay=1e-4   # L2 penalty coefficient
)

Effective loss = task_loss + weight_decay × sum(w²)
                                             ↑
                              penalty for large weights

Prevention Strategy 5: Early Stopping

Stop training when validation loss stops improving. This prevents the model from continuing to memorize the training set after it has already reached its best generalization. Early stopping is covered in detail in topic 37.

best_val_loss = infinity
patience = 10   # stop after 10 epochs without improvement
no_improve = 0

each epoch:
  if val_loss < best_val_loss:
    save model
    best_val_loss = val_loss
    no_improve = 0
  else:
    no_improve += 1
    if no_improve >= patience: stop training

Prevention Strategy 6: Batch Normalization

Batch normalization normalizes activations between layers, which has a mild regularizing effect and often reduces the need for aggressive dropout.

Prevention Strategy 7: Learning Rate Reduction

A learning rate that is too high can cause the model to overshoot the minimum and land on configurations that overfit. Use a learning rate scheduler to reduce the learning rate as training progresses.

Combining Strategies

Typical production recipe:
  ✓ Data augmentation (images) or input noise (tabular)
  ✓ Appropriate model size for dataset size
  ✓ Dropout in fully connected layers (p=0.3–0.5)
  ✓ weight_decay=1e-4 in Adam or AdamW
  ✓ BatchNorm after conv/linear layers
  ✓ Early stopping with patience=10–20
  ✓ Learning rate scheduler (ReduceLROnPlateau or CosineAnnealing)

Bias-Variance Tradeoff

Underfitting (high bias):
  Model too simple → cannot learn the pattern
  Both train and val loss are high
  Fix: larger model, more layers, more capacity

Overfitting (high variance):
  Model too complex → memorizes training data
  Train loss low, val loss high
  Fix: regularization, dropout, more data, smaller model

Goal: find the sweet spot where both losses are low.

Summary

Overfitting produces low training loss but high validation loss. The model memorizes instead of generalizing. Prevention strategies include: collecting more data or augmenting existing data, reducing model size, applying dropout, using L2 weight decay in the optimizer, adding batch normalization, scheduling the learning rate, and early stopping. In practice, combining two or three of these strategies produces the best results. Always monitor both training and validation loss to detect overfitting early.