PyTorch Dropout

Dropout is a regularization technique that randomly sets a fraction of neuron outputs to zero during each forward pass of training. This prevents neurons from co-adapting — relying on each other too specifically — and forces the network to build multiple independent pathways to the correct answer. The result is a model that generalizes better to unseen data.

How Dropout Works

Training:
  Hidden layer outputs: [0.8,  1.2, -0.5,  0.3,  1.7]
  Dropout mask (p=0.5): [  1,    0,    1,    0,    1]   ← random 50% zeroed
  After dropout:        [0.8,  0.0, -0.5,  0.0,  1.7]   ← and scaled up

  Remaining values are scaled by 1/(1-p) = 2.0
  to keep the expected sum the same.

Inference:
  Dropout is disabled.
  All neurons are active.
  No scaling applied (already baked into training values).

The Lottery Ticket Intuition

Think of a neural network as a lottery with many tickets. Each neuron is a ticket. Dropout forces the network to win using random subsets of tickets each round. Eventually, each small subset must individually learn to produce correct results. The final network, using all tickets together, is far more robust than one that learned to rely on a specific small group.

Adding Dropout to a Model

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 512),
    nn.ReLU(),
    nn.Dropout(p=0.5),        # drop 50% of activations
    nn.Linear(512, 256),
    nn.ReLU(),
    nn.Dropout(p=0.3),        # drop 30% of activations
    nn.Linear(256, 10)
)

Dropout in train() vs eval() Mode

Dropout must be active during training and disabled during evaluation. PyTorch handles this automatically when you call model.train() and model.eval().

model.train()   # dropout ON — neurons randomly zeroed
output = model(x)

model.eval()    # dropout OFF — all neurons active
with torch.no_grad():
    prediction = model(x_test)

Forgetting to call model.eval() before validation is one of the most common PyTorch bugs. It causes validation predictions to be noisier than they should be — different every time you run them.

Choosing the Dropout Rate

p = 0.5 (50%) → most common for large fully connected layers
p = 0.3 (30%) → lighter regularization; use for smaller layers
p = 0.2 (20%) → very light; use for conv layers or small networks
p = 0.0       → no dropout

Rule of thumb:
  → Large layers (512+): p = 0.4 – 0.5
  → Medium layers (128–256): p = 0.3 – 0.4
  → Small layers (<128): p = 0.1 – 0.2
  → Convolutional layers: p = 0.1 – 0.25 (or skip)

Spatial Dropout for CNNs

Standard Dropout drops individual values from a feature map. Spatial Dropout (nn.Dropout2d) drops entire channels. This is more effective for CNNs because adjacent pixels in a feature map are strongly correlated — dropping one pixel is not very disruptive, but dropping an entire channel forces the network to not rely on specific learned filters.

cnn_block = nn.Sequential(
    nn.Conv2d(32, 64, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.Dropout2d(p=0.2)   # drops entire feature map channels
)

Dropout as an Ensemble

Each training step activates a different random subset of neurons — effectively training a different "thinned" network each time. At test time, using all neurons simultaneously approximates averaging over all these different thinned networks. This ensemble effect is a large part of why dropout improves generalization.

Training:
  Step 1: uses neurons {1,3,5,7}
  Step 2: uses neurons {2,4,6,8}
  Step 3: uses neurons {1,2,5,8}
  ...
  Each step trains a different "sub-network"

Evaluation:
  Uses all neurons → approximates average of all sub-networks

Monte Carlo Dropout for Uncertainty Estimation

You can keep dropout active at inference time and run multiple forward passes. The variance in predictions across runs estimates the model's uncertainty. This technique is called MC Dropout.

model.train()   # keep dropout on during inference

predictions = []
with torch.no_grad():
    for _ in range(30):    # 30 stochastic forward passes
        pred = model(x_test)
        predictions.append(pred)

mean_pred = torch.stack(predictions).mean(0)
std_pred  = torch.stack(predictions).std(0)   # uncertainty estimate
print("Mean:", mean_pred)
print("Uncertainty:", std_pred)

Summary

Dropout randomly zeroes a fraction of activations during training, preventing neurons from co-adapting and improving generalization. Use nn.Dropout(p) after linear layers and nn.Dropout2d(p) after conv layers. Always call model.train() before training and model.eval() before evaluation — PyTorch automatically enables and disables dropout based on the mode. Common dropout rates are 0.5 for large fully connected layers and 0.2 for conv layers. Combine dropout with weight decay for the strongest regularization effect.