PyTorch Gradient

Gradient computation is the mathematical process that tells a neural network how to improve. Each gradient is a number that says: "if you change this weight slightly, the error will change by this much." PyTorch computes every gradient automatically using the chain rule of calculus.

What a Gradient Means in Plain Terms

Imagine you are standing on a hilly landscape. Your goal is to reach the lowest point (smallest error). The gradient tells you the slope of the hill beneath your feet — how steep it is and which direction is downhill. You take a step in the downhill direction. Repeat this thousands of times and you reach the valley (minimum error).

High Error
    │
    │  ← gradient points uphill
    │
    ▼  ← you step downhill (negative gradient direction)
    │
Low Error  ← goal

Computing Gradients with backward()

import torch

w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

# Simple linear model: y_hat = w * x + b
x = torch.tensor(2.0)
y = torch.tensor(6.0)    # target

y_hat = w * x + b
loss = (y_hat - y) ** 2  # squared error

loss.backward()           # compute gradients

print(w.grad)   # d(loss)/dw
print(b.grad)   # d(loss)/db

Tracing the Math

y_hat = w * x + b = 1 * 2 + 0 = 2
loss = (2 - 6)² = (-4)² = 16

d(loss)/d(y_hat) = 2 * (y_hat - y) = 2 * (2-6) = -8
d(y_hat)/dw = x = 2
d(y_hat)/db = 1

d(loss)/dw = d(loss)/d(y_hat) * d(y_hat)/dw = -8 * 2 = -16
d(loss)/db = d(loss)/d(y_hat) * d(y_hat)/db = -8 * 1 = -8

PyTorch computes exactly these numbers through autograd. You don't need to know the calculus — but seeing it once helps you trust what autograd is doing.

Gradient for a Vector of Weights

import torch

W = torch.tensor([0.5, -0.3, 0.8], requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0])

# dot product followed by squaring
output = (W * x).sum() ** 2

output.backward()
print(W.grad)   # gradient vector — one value per weight

Retaining the Graph

By default, PyTorch destroys the computation graph after calling backward() to save memory. If you need to call backward() more than once on the same graph, use retain_graph=True.

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x ** 3

y.backward(retain_graph=True)   # first backward
print(x.grad)   # 3x² = 27

x.grad.zero_()  # clear gradient
y.backward()    # second backward (same graph, now freed)
print(x.grad)   # 27 again

Gradient with Respect to Multiple Outputs

When a model produces a vector output (not a scalar), you cannot call .backward() directly. You must first reduce the output to a scalar — typically by calling .sum() or .mean() — or pass a gradient tensor as an argument.

import torch

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x ** 2   # y is a vector

# Option 1: reduce to scalar
y.sum().backward()
print(x.grad)   # tensor([2., 4., 6.])  ← d(sum of y)/dx = 2x

# Option 2: pass external gradient
x.grad.zero_()
y.backward(torch.ones_like(x))   # equivalent to y.sum().backward()
print(x.grad)   # tensor([2., 4., 6.])

Gradient Clipping

In very deep networks, gradients can grow explosively large during backpropagation. This is called the exploding gradient problem. Gradient clipping caps gradients at a maximum value to keep training stable.

import torch
import torch.nn as nn

model = nn.Linear(10, 1)
# ... training step ...
# After loss.backward():
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Checking Gradient Values

During training, you can inspect gradient values to diagnose training problems. Very small gradients signal vanishing gradients. Very large gradients signal exploding gradients. Both cause models to train poorly.

import torch.nn as nn

model = nn.Linear(10, 1)
# After loss.backward():
for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: grad mean = {param.grad.mean():.4f},  max = {param.grad.abs().max():.4f}")

Summary

Gradient computation uses the chain rule to calculate how much each weight contributed to the model's error. PyTorch's autograd engine does this automatically when you call loss.backward(). Gradients accumulate across calls, so clear them with zero_grad() before each training step. Use retain_graph=True if you need multiple backward passes on the same graph. Gradient clipping prevents exploding gradients in deep networks. Inspecting gradient values helps diagnose common training problems.