Bias-Variance Tradeoff
What is the bias-variance tradeoff?
A model's test error splits into bias (error from being too simple to capture the real pattern) and variance (error from being so flexible it memorizes noise). Add complexity and bias falls while variance rises — so you tune complexity to minimize their sum, not either one alone.
Too simple. Misses the signal and does poorly on both train and test — like fitting a straight line to a curve.
Too flexible. Nails the training data but chases noise, so test error is far worse than train error.
Lowering one usually raises the other. The goal is the lowest total error, where the two balance.
- “What's irreducible error?” — Noise inherent in the data/labels; no model can remove it. It sets the floor under the U-curve.
How do you diagnose high bias vs high variance?
Look at the gap between training and validation error. Both high with a small gap → bias problem. Low train but a large gap → variance problem. That one check tells you which lever to pull.
- Train error high — can't even fit what it has seen.
- Train ≈ test error — both bad, gap small.
- Fix: bigger model, more features, less regularization, train longer.
- Train error low — it memorized the training set.
- Test error ≫ train error — large gap.
- Fix: more data, regularization, simpler model, dropout, early stopping.
The classic picture is the dartboard: bias is how far your cluster sits from the bullseye; variance is how spread out your throws are. Low bias + low variance — tight and centered — is the goal.
Bias shifts the whole cluster off-center; variance scatters the throws. You want the top-left: tight and on target.
How do you reduce bias and variance?
They pull in opposite directions, so reach for different levers depending on which one dominates — diagnose first, then act.
- Use a more flexible model (deeper tree, more layers).
- Add features or better feature engineering.
- Reduce regularization; train longer.
More data mainly reduces variance; bias barely moves. Regularization trades a little bias for a big drop in variance. Pick the lever that targets your actual problem.
- “How does more training data affect each?” — It mainly reduces variance; bias barely moves.
- “Where do random forests sit?” — Low bias from deep trees, and bagging averages away the variance. (See Ensemble Methods.)
- “Does regularization raise bias or variance?” — It raises bias slightly to cut variance a lot. (See Overfitting & Regularization.)
How do learning curves help diagnose this?
A learning curve plots train and validation error as the training-set size grows. Its shape tells you which problem you have — and crucially, whether more data will help.
- Both curves plateau at a high error, close together.
- More data won't help — you need a richer model.
- Low train error with a large, persistent gap to validation.
- More data narrows the gap — it will help.
What is double descent?
A modern wrinkle on the classic U-curve. As model size approaches and then passes the point of exactly fitting the training data (the interpolation threshold), test error can peak near that threshold and then fall again — so very large, over-parameterized models can generalize well despite near-zero training error.
- “Does this kill the tradeoff?” — No — the classic intuition still holds in the under-parameterized regime; double descent shows it's incomplete for huge models.