Batch Normalization
What does batch normalization do?
Batch norm normalizes a layer's activations across the current mini-batch — subtract the batch mean, divide by the batch standard deviation — so each feature has roughly zero mean and unit variance, then applies a learnable scale and shift.
Normalize: x̂ = (x − μ_batch) / √(σ²_batch + ε), then transform: y = γ·x̂ + β, where γ and β are learned.
Why does it help training?
It keeps activations in a stable range layer to layer, which lets you use higher learning rates, makes training less sensitive to initialization, and adds a little noise that acts as mild regularization.
Smoother optimization landscape; faster, more reliable convergence.
- “Isn't it about internal covariate shift?” — That was the original explanation; later work argues the bigger effect is smoothing the loss landscape. Either way it empirically speeds training.
What are gamma and beta for?
Forcing every layer to mean 0 / variance 1 can be too restrictive. The learnable γ (scale) and β (shift) let the network undo the normalization if that's optimal — so it never loses representational power.
- “Do you still need a bias before BN?” — No — β absorbs the bias, so the preceding layer's bias is redundant.
Why do train and inference behave differently?
At training time it uses the current batch's mean and variance. At inference you may have a single example, so it uses running averages of mean/variance collected during training instead.
- “Why does small batch size hurt BN?” — Batch statistics become noisy and unreliable, which is one reason layer/group norm are used when batches are tiny.
- “Forget to switch to eval mode?” — Inference then uses batch stats, giving unstable, batch-dependent predictions — a common bug.
Batch norm vs layer norm — when to use which?
They normalize along different axes. Batch norm normalizes each feature across the batch; layer norm normalizes each sample across its features — independent of batch size.
- Great for CNNs / vision with decent batch sizes.
- Depends on batch statistics; awkward for variable-length sequences.
- Independent of batch size; same at train and test.
- Standard in transformers and RNNs.