Vanishing & Exploding Gradients
What are vanishing and exploding gradients?
As backprop multiplies many small or large derivatives across layers, the gradient can shrink toward zero (vanishing → early layers stop learning) or grow toward infinity (exploding → unstable, NaN losses). Deep networks are most at risk.
Multiplying many sub-1 derivatives makes the gradient shrink exponentially toward the input layer — so the earliest layers update almost not at all.
Why do they happen?
Backprop multiplies a derivative at each layer. If those terms are consistently < 1, the product decays to zero; if consistently > 1, it explodes. Saturating activations (sigmoid/tanh) and poor weight initialization make it worse.
- “Why are sigmoid/tanh bad here?” — Their derivatives max out at 0.25 (sigmoid) and saturate to 0 at the tails, so products shrink fast.
- “Why are RNNs especially affected?” — The same weight is applied at every time step, so the product spans the whole sequence length.
How do you fix vanishing gradients?
Keep the per-layer derivative near 1 so signals survive the trip back.
Derivative is exactly 1 for positive inputs — no saturation-driven decay. (See Activations.)
Scale initial weights so activation variance is preserved layer to layer.
Keeps each layer's inputs well-scaled, stabilizing gradient magnitudes.
- “How do LSTMs help?” — Their cell-state gates let gradients flow across time almost unchanged. (See RNNs, LSTM & GRU.)
How do you fix exploding gradients?
Cap or shrink the gradient so a single huge step can't destabilize training.
Gradient clipping (rescale gradients above a threshold) is the go-to, especially for RNNs. Also help: smaller learning rate, weight regularization, and batch norm. A loss that suddenly becomes NaN is the classic exploding-gradient symptom.
- “Clip by value or by norm?” — Clipping by global norm is usually preferred — it preserves the gradient's direction while capping its size.