Deep Learning Weight Initialization
Every network starts with weights, and those starting numbers shape the whole training journey. A good start lets learning move smoothly. A poor start can freeze progress or send numbers spiralling out of control. Weight initialization is the art of choosing good starting values.
Why Starting Values Matter
Imagine a hiking group that begins a trip from different spots on a mountain. A group that starts near a clear trail reaches the valley quickly. A group that starts in a thick forest wastes hours. Starting weights work the same way. They decide where on the error landscape training begins.
Bad Idea 1: All Zeros
Setting every weight to zero looks neat, but it breaks learning. Every neuron in a layer computes the same output and receives the same gradient. All neurons then update in the same way and stay identical forever.
Layer with zero weights: Neuron A: output 0 gradient 0.3 new weight 0.3 Neuron B: output 0 gradient 0.3 new weight 0.3 Neuron C: output 0 gradient 0.3 new weight 0.3 Result: three copies of one neuron
Random values break this symmetry, so each neuron learns a different feature.
Bad Idea 2: Values Too Large or Too Small
Signals pass through many layers. Each layer multiplies the signal by its weights. Large weights make the signal grow at every layer until it explodes. Tiny weights shrink the signal until it vanishes.
Signal strength across 5 layers Weights too large: 1 -> 3 -> 9 -> 27 -> 81 -> 243 (explodes) Weights too small: 1 -> 0.3 -> 0.09 -> 0.03 -> 0.008 (vanishes) Weights balanced: 1 -> 1.1 -> 0.9 -> 1.0 -> 1.0 (stable)
Deep networks need a balanced start so the signal keeps a healthy size from the first layer to the last.
Smart Initialization Methods
Xavier (Glorot) Initialization
Xavier initialization sets the weight range using the number of inputs and outputs of a layer. It suits Sigmoid and Tanh activations. Wider layers receive smaller weights, which keeps the signal steady.
He Initialization
He initialization scales weights using the number of inputs only, with a slightly larger spread. It suits ReLU activations, because ReLU switches off about half of its neurons and the extra spread compensates.
Small Random Values
Simple random draws from a narrow bell curve work for shallow networks. Deep networks need the scaled methods above.
Choosing a Method
| Activation Function | Recommended Initialization |
|---|---|
| ReLU and Leaky ReLU | He |
| Sigmoid | Xavier |
| Tanh | Xavier |
| Softmax output layer | Xavier |
What About Biases
Biases usually start at zero. Weights already break the symmetry between neurons, so biases do not need random values.
Initialization in Code
# Keras layers.Dense(64, activation="relu", kernel_initializer="he_normal") # PyTorch nn.init.kaiming_normal_(layer.weight)
Most frameworks pick a sensible default, but knowing the choice helps you fix a model that refuses to learn.
Warning Signs of Poor Initialization
- Loss stays flat from the first epoch
- Loss jumps to a huge number or becomes NaN
- Early layers show gradients near zero
Key Terms
- Initialization - choosing the starting weight values
- Symmetry - neurons that behave identically
- Exploding signal - values that grow larger at every layer
- Vanishing signal - values that shrink toward zero
- Xavier and He - scaled methods matched to the activation function