Deep Learning Batch Normalization

Batch normalization keeps the numbers inside a network tidy while it trains. It rescales the values flowing between layers so they stay in a comfortable range. Networks with this technique train faster, tolerate higher learning rates, and often generalize better.

The Problem It Solves

Each layer learns from the output of the layer before it. While training, earlier layers change their weights constantly, so later layers keep receiving inputs with shifting scales. Picture a student who changes classrooms every hour, and each teacher uses a different grading scale. Learning slows because nothing stays steady.

Without normalization:
Layer 1 output: values around 0 to 1
Layer 2 output: values around 0 to 50
Layer 3 output: values around -200 to 400
(each layer faces a different scale)

With normalization:
Every layer output: values centred near 0 with a similar spread

How Batch Normalization Works

The method processes one mini-batch of data at a time. It follows four steps.

Step 1: Find the Batch Mean

Average each feature across all examples in the batch.

Step 2: Find the Batch Variance

Measure how widely the values spread around that mean.

Step 3: Normalize

Subtract the mean and divide by the spread. The values now centre on zero with a standard spread.

Step 4: Scale and Shift

Multiply by a learnable number called gamma and add a learnable number called beta. The network can restore any scale it finds useful, so normalization never limits its power.

A Small Worked Example

Batch values for one neuron: 2, 4, 6, 8

Mean     = 5
Spread   = about 2.24

Normalized values:
  (2 - 5) / 2.24 = -1.34
  (4 - 5) / 2.24 = -0.45
  (6 - 5) / 2.24 =  0.45
  (8 - 5) / 2.24 =  1.34

Where to Place It

Most networks insert a batch normalization layer between a dense or convolution layer and its activation function.

Input -> Dense -> BatchNorm -> ReLU -> Dense -> BatchNorm -> ReLU -> Output

Benefits

• Faster training, often needing far fewer epochs
• Higher learning rates without instability
• Less dependence on careful weight initialization
• Mild regularization that reduces overfitting

Training Versus Prediction

During training, the layer uses the statistics of the current batch. During prediction, a single example has no batch to average. The layer instead uses running averages collected throughout training.

Training:    use mean and variance of the current batch
Prediction:  use stored running mean and running variance

Frameworks switch modes with a single command. Forgetting the switch produces strange results at prediction time.

Limits and Alternatives

Very small batches give noisy statistics, which weakens the method. Sequence models and Transformers prefer layer normalization, which averages across the features of one example rather than across the batch.

Method	Averages Across	Common Use
Batch normalization	Examples in a batch	Image networks
Layer normalization	Features of one example	Transformers and sequence models

Practical Tips

• Use batch sizes of 16 or more so the statistics stay stable
• Raise the learning rate after adding the layer and compare the results
• Call the evaluation mode of your framework before predicting
• Skip the layer in the final output layer of a network

Key Terms

• Normalization - rescaling values to a standard centre and spread
• Mini-batch - a small group of examples processed together
• Gamma and Beta - learnable scale and shift values
• Running average - statistics stored for use at prediction time