Ensemble Methods
Bagging vs boosting — what's the difference?
Both combine many base models into a stronger one. Bagging trains them in parallel on random samples and averages their votes — mainly cutting variance. Boosting trains them in sequence, each fixing the previous one's mistakes — mainly cutting bias.
Bagging (top) trains independent models in parallel and averages them to cut variance. Boosting (bottom) chains models so each fixes the last one's errors, cutting bias.
- Training: parallel, independent on bootstrap samples.
- Goal: reduce variance by averaging.
- Weak learners: deep, low-bias (overfit-prone) trees.
- Robust to: noise and outliers.
- Training: sequential, each fixes prior errors.
- Goal: reduce bias by error-correcting.
- Weak learners: shallow, high-bias stumps.
- Sensitive to: noise (it chases hard points).
- “What is bootstrapping?” — Sampling the training set with replacement to make each model's dataset.
- “Which parallelizes?” — Bagging trivially; boosting is sequential (though libraries parallelize within each tree).
How does a Random Forest work?
A Random Forest is bagging applied to decision trees, with one extra twist: at each split it considers only a random subset of features. That decorrelates the trees so averaging them cuts variance far more than bagging alone.
Each tree trains on a different sample drawn with replacement from the data.
At every split, only a random subset of features is eligible — so no single strong feature dominates every tree.
Classify by majority vote, regress by averaging. Independent errors cancel out.
Averaging only cuts variance if the trees make different mistakes. Feature subsetting forces that diversity — it's what makes a forest more than plain bagging. See the full Random Forest article.
- “Why is Random Forest more than bagging?” — It also randomizes the feature subset at each split to decorrelate trees.
- “What's out-of-bag error?” — Each tree's validation on the ~37% of rows it didn't sample — a free CV estimate.
Gradient Boosting and XGBoost — how do they work?
Gradient boosting builds shallow trees in sequence, where each new tree fits the residual errors (the negative gradient of the loss) of the ensemble so far. XGBoost is an optimized, regularized implementation that dominates tabular ML.
- Reweights misclassified points so the next stump focuses on them.
- Combines stumps with weights based on their accuracy.
- Fits each new tree to the residual gradient of the loss.
- Adds regularization, shrinkage (learning rate), and clever tree-building. See XGBoost.
Too many sequential rounds memorizes noise. Control it with a small learning rate, shallow tree depth, and early stopping on a validation set.
- “Gradient boosting vs AdaBoost?” — AdaBoost reweights points; gradient boosting fits each tree to the residual gradient.
Which should you reach for?
Bagging for stability and noisy data; boosting for maximum accuracy on clean data when you can afford to tune.
When your base model overfits, data is noisy, or you want easy parallel training with little tuning. Random Forest is a strong default.
When you need top performance on clean tabular data and can tune carefully. XGBoost/LightGBM dominate competitions.
A meta-model learns how to best combine diverse base models' predictions — the third major ensemble.