CV Training and Overfitting

Training teaches a neural network to improve through repeated practice. The model sees an image, guesses a label, measures how wrong the guess was, and adjusts its internal numbers. After thousands of rounds, the guesses sharpen. A good training run needs the right settings and honest monitoring.

The Training Loop in Layman Terms

Imagine a student working through flashcards. The student answers a card, checks the back, and notes the mistake. Repeated study sessions improve recall. A network follows the same pattern with images in place of flashcards.

 +-> Take a batch of images
 |        |
 |        v
 |   Predict labels (forward pass)
 |        |
 |        v
 |   Measure error (loss)
 |        |
 |        v
 |   Adjust weights (backward pass + optimizer)
 |        |
 +--------+      repeat for every batch, every epoch

Key Training Terms

• Epoch: one full pass through the training set
• Batch size: how many images the model sees before each weight update
• Loss: a number that shows how wrong the predictions are, where lower is better
• Learning rate: the size of each adjustment step
• Optimizer: the rule that turns errors into adjustments, such as SGD or Adam

Learning Rate Intuition

Picture walking down a foggy hill toward the valley floor. Tiny steps take forever. Giant leaps overshoot the valley and land on the opposite slope. A moderate step size reaches the bottom steadily. Values near 0.001 work for Adam, and 0.01 works for SGD with momentum on many tasks.

A Minimal PyTorch Loop

import torch
import torch.nn as nn

model = model.to("cuda" if torch.cuda.is_available() else "cpu")
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for epoch in range(10):
    model.train()
    for images, labels in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(images), labels)
        loss.backward()
        optimizer.step()

    model.eval()
    correct = 0
    with torch.no_grad():
        for images, labels in val_loader:
            correct += (model(images).argmax(1) == labels).sum().item()
    print("Epoch", epoch, "val accuracy", correct / len(val_loader.dataset))

The model switches between training mode and evaluation mode, because layers such as dropout behave differently in each.

Reading Loss Curves

Plot training loss and validation loss after every epoch. The two curves tell the health story.

Healthy run                    Overfitting run
loss                           loss
 |\                             |\
 | \__ train                    | \__ train  (keeps falling)
 |  \__ val                     |    \___
 |     \___                     |  val  ____/  (turns upward)
 +------------ epochs           +------------ epochs
Both fall together             Gap grows after epoch 8

What Overfitting Means

Overfitting happens when a model memorizes training photos without learning general patterns. A student who memorizes last year's exam answers scores perfectly on that exam and fails a new one. A model with a big gap between training accuracy and validation accuracy shows the same symptom.

Underfitting

Underfitting is the opposite problem. Both curves stay high, and the model lacks the capacity or training time to learn the task.

Remedies for Overfitting

Remedy               How it helps
-------------------  -------------------------------------------
More data            more variety to learn from
Data augmentation    creates new views of existing images
Dropout              randomly hides neurons during training
Weight decay         discourages extreme weight values
Early stopping       halts training when validation loss rises
Smaller model        less room to memorize
Transfer learning    starts from general features

best = float("inf")
patience, wait = 5, 0
for epoch in range(100):
    val_loss = run_epoch()          # your validation function
    if val_loss < best:
        best, wait = val_loss, 0
        torch.save(model.state_dict(), "best.pt")
    else:
        wait += 1
        if wait >= patience:
            break

Early stopping saves the best weights and stops after five epochs without improvement.

Fixing Underfitting

• Train for more epochs
• Use a larger or deeper model
• Raise the learning rate slightly
• Check that labels are correct and images load properly

Debugging Checklist

• Overfit a tiny set of 20 images first; the loss should drop near zero
• Normalize images with the same mean and standard deviation used in pretraining
• Verify that labels match images by plotting a few samples
• Track the learning rate, loss, and accuracy in a log file

Practice Exercise

Train a small CNN on CIFAR-10 for 20 epochs without augmentation and plot both curves. Repeat the run with flips, crops, and dropout. Compare the gap between the two curves in each experiment.

Choosing Batch Size and Epochs

Small batches add noise to each update and often generalize well, while large batches run faster on powerful hardware. Values such as 16, 32, and 64 suit most beginner projects. The right number of epochs depends on the dataset: small projects often converge within 10 to 30 epochs, and early stopping removes the guesswork.

Learning Rate Schedules

Many teams lower the learning rate as training progresses. Large steps early on make fast progress, and small steps later fine-tune the weights. Cosine decay and step decay both work well and take one line of code in PyTorch.