Loss Functions
What is a loss function and how is it chosen?
The loss function is the single number training minimizes — it measures how wrong the model's predictions are. The choice encodes what “wrong” means for your task, so it's driven by the problem type and how you want errors penalized.
- “Loss vs metric?” — The loss must be differentiable to train on; the metric (accuracy, F1) is what you actually care about and may not be differentiable.
MSE vs MAE vs Huber for regression?
They differ in how harshly they punish large errors — i.e. how sensitive they are to outliers.
Smooth, penalizes big misses hard — sensitive to outliers. Optimizes the mean.
Robust to outliers; optimizes the median. Gradient is constant, harder near zero.
Quadratic for small errors, linear for large — smooth yet outlier-tolerant.
What loss do you use for classification?
Cross-entropy (log loss). It measures the distance between the predicted probability distribution and the true labels, heavily penalizing confident wrong predictions.
Why not use MSE for classification?
Two reasons. MSE on top of a sigmoid gives a non-convex, flat-gradient loss — when the model is confidently wrong, the gradient is tiny, so it learns slowly. And MSE doesn't match the probabilistic objective the way cross-entropy does.
- “What does cross-entropy fix?” — Its gradient is large exactly when the prediction is confidently wrong, so the model corrects fast.
How do you handle class imbalance in the loss?
When one class dominates, plain cross-entropy is swamped by the majority class. Reweight the loss so the minority class matters.
Weight each class inversely to its frequency.
Down-weights easy, well-classified examples — popular for detection.
Oversample minority / undersample majority as an alternative.