PyTorch Autograd

Autograd is PyTorch's automatic differentiation engine. It watches every operation you perform on tensors and records them. When you call .backward(), it uses those records to compute gradients automatically — saving you from doing calculus by hand.

Why Gradients Matter

A neural network learns by adjusting its weights to reduce errors. Gradients tell you in which direction and by how much to adjust each weight. Without gradients, training is impossible. Calculus students compute derivatives by hand. Autograd computes them automatically for any computation, no matter how complex.

The Computation Graph

Every time you perform a math operation on a tensor with requires_grad=True, PyTorch builds a node in a computation graph. This graph tracks which operations produced each value. When you call .backward(), PyTorch walks this graph backwards and computes gradients.

x = 3.0  (leaf node, requires_grad=True)
     │
     ▼
y = x²        (square operation)
     │
     ▼
z = y + 5     (add operation)
     │
     ▼
z.backward()  ← PyTorch traces back through graph
     │
     ▼
x.grad = dz/dx = 2x = 6

Basic Autograd Example

import torch

x = torch.tensor(3.0, requires_grad=True)

y = x ** 2         # y = x²
z = y + 5          # z = x² + 5

z.backward()       # compute gradients

print(x.grad)      # tensor(6.)  ← dz/dx = 2x = 2×3 = 6

Breaking Down the Calculation

Calculus tells us that the derivative of x² with respect to x is 2x. At x=3, that gives 6. And the derivative of z = y + 5 with respect to y is 1. By the chain rule, the gradient of z with respect to x is 1 × 2x = 2×3 = 6. Autograd computed this instantly and stored it in x.grad.

Multi-Variable Gradients

import torch

a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(3.0, requires_grad=True)

c = a * b + a ** 2   # c = ab + a²

c.backward()

print(a.grad)    # dc/da = b + 2a = 3 + 4 = 7
print(b.grad)    # dc/db = a = 2

How the Graph Is Built Step by Step

  a=2               b=3
   │                 │
   ├──── a*b ────────┤
   │       │
   │       ▼
   │     term1 = ab = 6
   │
   ├──── a² = 4
   │       │
   │       ▼
   │     term2 = a² = 4
   │
   └─────────────────→ c = term1 + term2 = 10

c.backward():
  dc/da = b + 2a = 3 + 2(2) = 7
  dc/db = a = 2

Clearing Gradients with zero_grad

Gradients accumulate. Every time you call .backward(), PyTorch adds to the existing gradient value — it does not replace it. In a training loop, call optimizer.zero_grad() before each backward pass to clear the accumulated gradients from the previous step.

import torch

x = torch.tensor(2.0, requires_grad=True)

for _ in range(3):
    y = x ** 2
    y.backward()
    print(x.grad)    # 4, then 8, then 12 — keeps accumulating!
    x.grad.zero_()   # must clear before next step

Disabling Gradient Tracking

During inference (making predictions without training), you don't need gradients. Disabling them with torch.no_grad() reduces memory usage and speeds up computation.

import torch
import torch.nn as nn

model = nn.Linear(3, 1)
x = torch.rand(5, 3)

with torch.no_grad():
    predictions = model(x)
    print(predictions)   # no gradient computation, runs faster

The Role of Autograd in Training

Full Training Step:

1.  model(X)        ← forward pass: builds computation graph
2.  loss_fn(pred, y) ← calculates the error
3.  zero_grad()     ← clears old gradients
4.  loss.backward() ← autograd computes all gradients
5.  optimizer.step() ← uses gradients to update weights
6.  repeat

Leaf Nodes vs Non-Leaf Nodes

A leaf node is a tensor you created directly — like x = torch.tensor(3.0, requires_grad=True). Non-leaf nodes are the results of operations on leaf nodes. After calling .backward(), PyTorch stores gradients only on leaf nodes by default. Intermediate results are discarded to save memory.

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2   # y is non-leaf (created by operation)

y.backward()

print(x.grad)    # tensor(6.)  ← leaf: has gradient
print(y.grad)    # None  ← non-leaf: gradient discarded

Summary

Autograd is PyTorch's automatic differentiation system. It builds a computation graph as you perform operations on tensors with requires_grad=True. Calling .backward() walks the graph in reverse and computes gradients using the chain rule. Gradients accumulate, so call zero_grad() before each backward pass. Use torch.no_grad() during inference to save memory and speed up prediction. Autograd is the engine that makes neural network training possible — without it, you would have to derive and code every gradient by hand.