Dropout
What is dropout and why does it help?
Dropout randomly switches off a fraction of neurons (typically 20–50%) on each training step. The network can't rely on any single neuron always being present, so it learns redundant, robust features instead of fragile co-adaptations. See the dropout explainer.
- “Is it a regularizer?” — Yes — it reduces overfitting by adding noise that prevents the network from memorizing the training set.
- “Typical rate?” — 0.5 for dense layers is classic; lower (0.1–0.2) for inputs and convolutional layers.
What happens at train vs test time?
Dropout is only active during training. At test time every neuron is on — you want the full, deterministic network for inference.
- Each step drops a different random subset of neurons.
- Forces redundancy; effectively trains a different thinned network each step.
- No dropout — all neurons active.
- No scaling at test time — activations were already scaled by 1/(1−p) during training (inverted dropout).
Why scale the activations (inverted dropout)?
If you drop a fraction p of neurons during training, the layer's expected sum is smaller than at test time. To keep the expected value consistent, you scale the surviving activations by 1/(1−p) during training — this is inverted dropout, so test-time code needs no change.
- “What if you didn't scale?” — Test-time activations would be systematically larger than training, shifting every downstream computation.
Why is dropout like an ensemble?
Each training step uses a different random sub-network, so over training you implicitly train a huge family of thinned networks that share weights. At test time, using all neurons (scaled) approximates averaging all those sub-networks — ensemble-like robustness from a single model, almost for free.
How does dropout interact with batch norm?
They can clash. Batch norm already regularizes and relies on activation statistics, which dropout's noise disrupts — so modern architectures often use batch norm instead of heavy dropout, or apply dropout only in the final dense layers.