Optimizers
What does momentum add to plain SGD?
Plain SGD steps in the direction of the current gradient. Momentum accumulates an exponentially-weighted average of past gradients — like a ball rolling downhill — so it accelerates along consistent directions and damps oscillations across ravines.
- “What problem does it fix?” — SGD zig-zags in steep, narrow valleys; momentum smooths the path and speeds convergence.
What are adaptive learning rates (RMSprop)?
Adaptive methods give each parameter its own effective learning rate, scaled down for parameters with large, frequent gradients. RMSprop divides the step by the square root of a running average of recent squared gradients (its root-mean-square), so rarely-updated weights still move.
How does Adam work?
Adam = momentum + RMSprop. It keeps a running mean of gradients (first moment, like momentum) and a running mean of squared gradients (second moment, like RMSprop), with bias correction for the early steps.
Momentum-style average smooths the update direction.
Squared-gradient average adapts the step size per parameter.
Robust out-of-the-box — a common reason Adam is the default.
- “Why bias correction?” — The moment estimates start at zero, so they're biased toward zero early; the correction rescales them in the first steps.
How important is the learning rate, and what are schedules?
The learning rate is the single most important hyperparameter. Too high and training diverges; too low and it crawls or gets stuck.
Common practice is to decay the rate over time (step, cosine) and sometimes warm up from a small value for the first steps — especially for transformers — to avoid early instability.
Weight decay vs L2 — and SGD vs Adam in practice?
With plain SGD, weight decay and L2 regularization are equivalent. With Adam they aren't — the adaptive scaling distorts L2, which is why AdamW (decoupled weight decay) is preferred.
- “Adam or SGD?” — Adam/AdamW converge fast and are forgiving (great for transformers). Well-tuned SGD with momentum often generalizes slightly better on vision tasks.