Hyperparameter Tuning
Parameters vs hyperparameters — what's the difference?
Parameters are learned from the data during training (the weights of a linear model, the splits of a tree). Hyperparameters are set before training and control how learning happens — they aren't learned by the optimizer.
Weights, biases, tree split points — fit by minimizing the loss.
Learning rate, tree depth, k in KNN, regularization strength, number of layers.
Try combinations, score each on validation data, keep the best.
Grid vs random vs Bayesian search?
They differ in how they explore the hyperparameter space and how efficiently they spend their budget.
- Tries every combination on a predefined grid.
- Exhaustive but explodes combinatorially with dimensions.
- Samples random combinations for a fixed budget.
- Usually beats grid — it covers important dimensions more finely.
Bayesian optimization (e.g. Optuna's default TPE sampler) builds a model of which regions look promising and focuses the budget there, instead of sampling blindly; Hyperband instead saves budget by stopping poorly performing runs early. Best when each training run is expensive.
- “Why does random often beat grid?” — Only a few hyperparameters usually matter; random search samples many distinct values along those, while grid wastes runs on unimportant ones.
How does cross-validation fit into tuning?
You score each hyperparameter combination with k-fold cross-validation so the estimate isn't dependent on one lucky split. The combination with the best average CV score wins, then you refit on all the training data.
- “Why not tune on a single validation set?” — A single split is noisy; CV averages over folds for a more stable estimate.
What is nested cross-validation and why use it?
If you tune and report performance on the same CV folds, the reported score is optimistic — you've indirectly fit to those folds. Nested CV uses an inner loop to tune and an outer loop to estimate generalization.
Inner loop: pick the best hyperparameters. Outer loop: evaluate that whole tuning procedure on data it never saw. The outer score is an unbiased estimate of real-world performance.
What are the common tuning pitfalls?
The biggest one is overfitting the validation set: try enough combinations and one will look great by chance.
Fit scalers/encoders within each fold, not before the split.
Tuning for accuracy on imbalanced data is misleading — use F1 / AUC / business metric.
Spend search budget on impactful hyperparameters (learning rate, depth) first.
- “How do you keep a truly honest final number?” — Hold out a test set that's used exactly once, after all tuning is finished.