Classification Metrics
Precision vs recall — what's the difference?
Precision asks: of everything I flagged positive, how much was actually positive? Recall asks: of all the real positives, how many did I catch? Precision punishes false alarms; recall punishes misses — and raising the decision threshold trades one for the other.
Precision is read down the “predicted positive” column (TP + FP). Recall is read across the “actually positive” row (TP + FN).
- False positives are costly. A wrong flag is expensive or harmful.
- Spam filter: better to let some spam through than junk a real email.
- False negatives are costly. A miss is dangerous.
- Cancer screening: better a false alarm than missing a real tumor.
- “What's a confusion matrix for multiclass?” — An N×N grid; compute precision/recall per class and average (macro / weighted).
What is the F1 score and when do you use it?
F1 is the harmonic mean of precision and recall: F1 = 2·P·R / (P+R). It's high only when both are high, so it's the single number to reach for when you can't pick a side — especially on imbalanced data.
The harmonic mean is dragged down by the smaller value, so you can't fake a good F1 by maxing one metric and ignoring the other.
Raise the probability cutoff and you predict positive less often: fewer false alarms (precision up) but more misses (recall down).
A weighted F1: β>1 favors recall, β<1 favors precision. Use it when one error genuinely costs more.
- “When is F1 better than ROC-AUC?” — On heavily imbalanced data where the positive class is rare.
ROC-AUC vs PR curve — which and when?
Both trace performance across every threshold instead of one. The ROC curve plots true-positive rate vs false-positive rate; the PR curve plots precision vs recall. On heavily imbalanced data, prefer the PR curve — ROC can look deceptively good.
A perfect model hugs the top-left (AUC = 1); the diagonal is random guessing (AUC = 0.5). AUC is the probability the model ranks a random positive above a random negative.
Balanced classes → ROC-AUC is fine. Rare positive class (fraud, disease) → the PR curve / average precision reflects real-world performance far better, because ROC's false-positive rate is diluted by the huge negative class.
- “What is the ROC / PR curve exactly?” — They trace metric tradeoffs across every threshold; AUC / average precision summarize them in one number.
Why not just use accuracy?
On imbalanced data, accuracy is a trap. With 99% negatives, a model that predicts “always negative” scores 99% accuracy while catching zero positives — useless. Precision and recall expose exactly that failure.
Always ask what the class balance is before trusting accuracy. For regression the analog metrics are RMSE / MAE (error magnitude) and R² (variance explained) — accuracy doesn't apply there at all.
- “How do you handle class imbalance?” — Resampling, class weights, or threshold tuning — then judge with precision/recall, not accuracy.
- “Regression metrics?” — MAE (robust to outliers), RMSE (penalizes large errors), R² (fraction of variance explained).