Cross-Validation & Data Splits
Why split data into train / validation / test?
You need an honest estimate of how the model does on data it has never seen. Train fits the model, validation tunes hyperparameters and picks models, and the test set is touched once at the very end for a final, untouched score.
The model learns its parameters here.
Choose hyperparameters and compare models. Re-used many times, so it gets optimistic.
The final unbiased estimate. Look at it repeatedly and it stops being honest.
- “Can you overfit the validation set?” — Yes — tuning against it repeatedly; that's why the final test set stays untouched.
- “Nested CV — why?” — An inner loop tunes hyperparameters, an outer loop estimates performance, so tuning doesn't leak into the score.
How does k-fold cross-validation work?
Split the data into k folds. Train on k−1 of them and validate on the one left out — then rotate so every fold is the validation set once. Average the k scores for a far more stable estimate than a single split.
- “What k?” — 5 or 10 is standard; bigger k = less bias, more compute and variance.
- “What is LOOCV?” — Leave-one-out: k = n. Nearly unbiased but expensive and high-variance.
When do you stratify, and what about time series?
Plain random splits can leave a fold with too few of a rare class. Stratified k-fold keeps each fold's class balance the same as the whole — essential for imbalanced classification. Time series needs special care entirely.
- Preserves class proportions in every fold.
- Use for imbalanced classification.
- Never shuffle — that leaks the future into the past.
- Always train on past, validate on future (rolling window).
What is data leakage and how do you prevent it?
Leakage is when information from outside the training fold sneaks in, giving a falsely high score that collapses in production. The classic trap: scaling or imputing on the full dataset before splitting.
Fit every preprocessing step (scaling, imputation, encoding, feature selection) inside the cross-validation loop, on the training fold only — then apply it to the validation fold. Wrap it all in a Pipeline so it can't leak.