Backpropagation
What is backpropagation?
Backprop is the algorithm that computes the gradient of the loss with respect to every weight. After the forward pass, it sweeps backward through the network, using the chain rule to assign each weight its share of the blame for the error.
One forward pass computes the loss; one backward pass hands every weight its gradient. Then gradient descent updates them.
- “Is backprop the same as gradient descent?” — No: backprop computes gradients; gradient descent uses them to update weights.
How does the chain rule make it efficient?
A deep network is a composition of functions, so the gradient is a product of local derivatives. Backprop computes each local derivative once and reuses intermediate results, so the whole gradient costs about the same as one forward pass — not exponential.
Instead of recomputing paths, backprop caches each layer's activation on the way forward and multiplies derivatives on the way back — dynamic programming on the computational graph. This is why training huge nets is feasible.
- “What's a computational graph?” — A DAG of operations; autodiff frameworks build it and apply the chain rule automatically.
- “Forward-mode vs reverse-mode autodiff?” — Backprop is reverse-mode, efficient when outputs (the scalar loss) are few and inputs (weights) are many.
Which loss function do you use?
The loss is what backprop differentiates. Match it to the task: cross-entropy for classification, MSE for regression.
- Pairs with softmax/sigmoid outputs.
- Punishes confident wrong predictions hard; strong gradients.
- Average squared error for continuous targets.
- Avoid on classification — gradients are weak through the sigmoid.
- “Why not MSE for classification?” — Combined with sigmoid it gives a non-convex surface and tiny gradients when very wrong; cross-entropy fixes both.
What can go wrong during backprop?
The gradient is a long product of terms, so it can shrink toward zero (vanishing) or blow up (exploding) across many layers — stalling or destabilizing training.
- “How do you fix vanishing/exploding gradients?” — ReLU, careful init, batch norm, residual connections, gradient clipping. (See Vanishing & Exploding Gradients.)
- “What is gradient checking?” — Numerically approximating gradients to verify your backprop implementation is correct.
- “Why zero the gradients each step?” — Frameworks accumulate gradients by default, so you reset them before each backward pass.