PyTorch requires_grad

The requires_grad attribute is a switch. When set to True on a tensor, PyTorch watches every operation involving that tensor and records the steps needed to compute gradients. When set to False, PyTorch ignores the tensor in gradient computations, saving both time and memory.

Default Behavior

By default, tensors do not track gradients. You must explicitly turn tracking on for tensors whose gradients you need — typically model weights and biases, not input data.

import torch

a = torch.tensor([1.0, 2.0])
print(a.requires_grad)   # False — default

b = torch.tensor([1.0, 2.0], requires_grad=True)
print(b.requires_grad)   # True

Why Inputs Usually Do Not Need requires_grad

In a neural network, you want to adjust the model's weights to reduce error — not the input data. The input data (your training examples) is fixed. So you set requires_grad=True on model parameters, not on data tensors.

Neural network:
  Input X          → requires_grad = False  (data, stays fixed)
  Weights W        → requires_grad = True   (learnable)
  Bias b           → requires_grad = True   (learnable)
  Predictions      → created by operations on W and b
  Loss             → scalar, used to compute gradients of W and b

Turning requires_grad On and Off

t = torch.tensor([1.0, 2.0, 3.0])

t.requires_grad_(True)    # in-place: turn on
print(t.requires_grad)    # True

t.requires_grad_(False)   # in-place: turn off
print(t.requires_grad)    # False

requires_grad Propagates Through Operations

If any tensor in a computation has requires_grad=True, the result of the operation also has requires_grad=True. This is how PyTorch builds the computation graph.

a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(3.0)         # requires_grad=False

c = a + b                     # c.requires_grad = True (because of a)
d = b * torch.tensor(5.0)     # d.requires_grad = False (b has no grad)

print(c.requires_grad)   # True
print(d.requires_grad)   # False

Freezing Layers with requires_grad

Setting requires_grad=False on specific model parameters freezes them — they will not be updated during training. This is fundamental to transfer learning, where you freeze early layers and train only the later ones.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 50),   # layer 0
    nn.ReLU(),
    nn.Linear(50, 1)     # layer 2
)

# Freeze layer 0
for param in model[0].parameters():
    param.requires_grad = False

# Check
for name, param in model.named_parameters():
    print(f"{name}: requires_grad={param.requires_grad}")

Output:
0.weight: requires_grad=False   ← frozen
0.bias:   requires_grad=False   ← frozen
2.weight: requires_grad=True    ← trainable
2.bias:   requires_grad=True    ← trainable

torch.no_grad() vs requires_grad=False

These two approaches both stop gradient tracking but they work differently.

requires_grad=False on a tensor:
  → that specific tensor never participates in gradient tracking
  → set once, applies permanently to that tensor

torch.no_grad() context manager:
  → temporarily disables gradient tracking for a block of code
  → used during inference or evaluation
  → does not permanently change any tensor's requires_grad setting

model.eval()   # sets model to evaluation mode

with torch.no_grad():
    predictions = model(X_test)
    # no gradient graph built here — faster and uses less memory

Checking Which Parameters Have Gradients

import torch.nn as nn

model = nn.Linear(5, 1)

# Count trainable parameters
total = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable parameters: {total}")

Summary

requires_grad tells PyTorch whether to track gradient information for a tensor. Set it to True on model weights and biases — the values you want to learn. Leave it False on input data, which stays fixed. Gradient tracking propagates: if one input to an operation requires gradients, the output does too. Set requires_grad=False on model layers to freeze them during transfer learning. Use torch.no_grad() during inference to temporarily suspend gradient tracking and run predictions faster.