Overfitting & Regularization
What is overfitting and how do you detect it?
Overfitting means the model learned the training data's noise instead of its signal — so training error is low but test error is high. You detect it by watching the gap between training and validation error: a large, growing gap is the tell.
Training error (green) keeps falling, but validation error (red) bottoms out and rises. The growing gap is overfitting; the orange line marks where early stopping would cut it off.
- Low train error, high test error — big gap.
- Performance gets worse as you train longer.
- Reach for: regularization, more data, simpler model.
- High train error too — small gap, both bad.
- The model can't capture the pattern at all.
- Reach for: bigger model, more features. (See Bias-Variance.)
- “Can you overfit the validation set?” — Yes, by tuning against it repeatedly; that's why you keep a final untouched test set.
How do you prevent overfitting?
Either give the model less room to memorize (simpler model, regularization, dropout, early stopping) or give it more to learn from (more / augmented data) — and always measure with a held-out set or cross-validation.
The most reliable fix. Real data or augmentation (crops, flips, noise) makes noise-fitting impossible.
Penalize large weights so the model stays smooth — see the next question.
Shallower trees, fewer layers/units, lower polynomial degree — less capacity to overfit.
Randomly drop units during training so the network can't rely on any single path.
Halt training when validation error stops improving — exactly the orange line above.
Not a fix itself, but k-fold CV gives a trustworthy estimate so you actually detect overfitting.
- “Why does more data help?” — It reduces variance; noise no longer has room to be memorized.
- “How does dropout work?” — Randomly zeroing units approximates training an ensemble of sub-networks.
L1 vs L2 regularization — what's the difference?
Both add a penalty on weight size. L1 (Lasso) penalizes the sum of absolute values and pushes some weights to exactly zero — built-in feature selection. L2 (Ridge) penalizes the sum of squares and shrinks all weights smoothly toward zero without eliminating them.
The penalty defines a constraint region the weights must stay inside. L1's is a diamond with corners on the axes; L2's is a smooth circle. The solution is where the loss contours first touch it — and a diamond gets touched at a corner, where a weight is zero.
The loss contours (blue) touch L1's diamond at a sharp corner on the axis — so a weight becomes exactly zero. They touch L2's smooth circle on its side, shrinking both weights but zeroing neither.
- Penalty: λ·Σ|w| (absolute values).
- Effect: drives weak weights to exactly zero → sparse model.
- Use when: you want feature selection or an interpretable model.
- Penalty: λ·Σw² (squares).
- Effect: shrinks all weights smoothly; keeps every feature.
- Use when: features are correlated or all carry some signal.
Elastic Net combines L1 and L2 — it gets sparsity from L1 while handling correlated features gracefully like L2. The constant pull of L1 (vs L2's vanishing pull near zero) is exactly the diamond's sharp corner.
- “What does λ control?” — Penalty strength: larger λ → more shrinkage → more bias, less variance.
- “Why scale features before regularizing?” — The penalty is scale-sensitive; standardize so it's applied fairly.
- “Is L2 the same as weight decay?” — In plain SGD, yes — L2 regularization equals weight decay.
What about underfitting?
Underfitting is the opposite failure: the model is too simple to capture the pattern, so it does poorly on both train and validation. It's a high-bias problem — and over-regularizing causes it.
- “How do you fix it?” — A more expressive model, better features, less regularization, or longer training.
What is Elastic Net?
Elastic Net blends the L1 and L2 penalties. It keeps L1's feature selection (sparse weights) while L2 handles correlated features gracefully — where pure L1 would arbitrarily pick one and drop the rest.
- “When prefer it?” — High-dimensional data with groups of correlated features, where you still want sparsity.