Gradient Descent & Optimization
What is gradient descent?
It minimizes a loss by repeatedly stepping downhill: compute the gradient (the slope of the loss w.r.t. the weights) and move a little in the opposite direction. Repeat and you roll toward the minimum.
w ← w − η · ∇L(w) — subtract the gradient scaled by the learning rate η. Steps shrink automatically as the slope flattens near the minimum.
Batch vs stochastic vs mini-batch — what's the difference?
They differ in how much data each step uses to estimate the gradient — trading accuracy of the step against speed and noise.
Smooth, accurate steps, but one step needs a full pass — slow and memory-heavy on big data.
Fast, frequent, noisy updates. The noise can even help escape shallow local minima.
The practical default — stable enough, fast, and GPU-friendly. Best of both.
- “What's an epoch vs a batch?” — A batch is one gradient step; an epoch is one full pass over the data (many batches).
Why does the learning rate matter so much?
The learning rate η sets the step size. Too small and training crawls; too large and it overshoots the minimum and can diverge. It's the single most important hyperparameter to get right.
- Overshoots and bounces across the valley.
- Loss oscillates or blows up to NaN.
- Too small: stable but painfully slow.
- Just right: steady, fast descent. Use schedules / warmup to adapt it.
- “Why scale features first?” — Unscaled features make the loss surface elongated, so descent zig-zags.
- “What is a learning-rate schedule?” — Decay η over time (step, cosine) so you take big steps early and fine ones near the minimum.
What do momentum and Adam add?
Plain gradient descent can crawl through flat regions and zig-zag across ravines. Momentum builds velocity in a consistent direction; Adam adapts a per-parameter learning rate. Both speed up and stabilize training.
Accumulates a moving average of past gradients — rolls through flats and damps oscillation.
Divides the step by the square root of a running average of squared gradients, so each parameter adapts.
Combines both with bias correction — the robust default for deep learning.
- “Local minima / saddle points?” — In high dimensions saddle points are the bigger problem; momentum and noise help escape them.