Decision Trees
How does a decision tree choose splits?
It greedily picks the feature and threshold that best separate the classes — the split giving the largest drop in impurity (the most information gain). Then it recurses on each branch until a stopping rule kicks in.
Each internal node asks a yes/no question about one feature. A branch becomes a leaf once it's pure enough; the path from root to leaf is the prediction rule.
- “How do trees handle regression?” — Splits minimize variance / MSE; each leaf predicts the mean of its samples.
- “How are missing values handled?” — Surrogate splits or sending them down the majority branch; some libraries learn a default direction.
Gini vs entropy — what's the difference?
Both measure how mixed a node's labels are; the tree chooses the split that reduces them most. In practice they give nearly identical trees — Gini is just slightly cheaper to compute.
1 − Σ pᵢ²— probability of mislabeling a random sample.- No logarithm, so faster — the CART/scikit-learn default.
- 0 = pure node, 0.5 = perfectly mixed (binary).
−Σ pᵢ·log₂ pᵢ— bits of uncertainty.- Information gain = parent entropy − weighted child entropy.
- 0 = pure, 1 = perfectly mixed (binary).
- “What's feature importance?” — Total impurity reduction a feature contributes across all its splits.
Why do trees overfit, and how does pruning help?
Left unchecked, a tree keeps splitting until every leaf is one sample — memorizing noise into a jagged boundary. Pruning and pre-stopping rules trade a little training accuracy for far better generalization.
Cap max_depth, require min_samples_leaf, or demand a minimum gain to split.
Grow fully, then remove branches that don't improve validation error (cost-complexity pruning).
A single tree is high-variance; averaging many tames it. See Ensemble Methods.
- “Why is a forest better than one tree?” — Bagging + feature subsetting decorrelates trees and averages away the variance.
What are the pros and cons?
- Highly interpretable — you can read the rules.
- No feature scaling needed; handles mixed types.
- Captures non-linear interactions automatically.
- High variance — small data changes reshape the tree.
- Greedy splits aren't globally optimal.
- Biased toward features with many levels.