Logistic Regression
Why does logistic regression use the sigmoid?
It computes a linear score z = w·x + b and squashes it through the sigmoid σ(z) = 1/(1+e⁻ᶻ), which maps any real number to a probability in (0, 1). A threshold (usually 0.5) turns that probability into a class.
- “What do the coefficients mean?” — Each is the change in log-odds per unit of the feature;
e^wis the odds ratio.
Why not just use linear regression for classification?
A straight line isn't bounded to [0, 1], so it predicts nonsensical probabilities like 1.7 or −0.3, and a single far-away point can tilt the whole line and move the boundary. The sigmoid + log-loss fix both.
- Outputs unbounded values — not valid probabilities.
- Outliers drag the line and shift the boundary.
- MSE on 0/1 labels is non-convex through the sigmoid.
- Sigmoid keeps outputs in (0, 1).
- Log-loss is convex — one global optimum.
- Coefficients have a clean log-odds interpretation.
What's the cost function, and why not MSE?
It minimizes log-loss (binary cross-entropy): it punishes a confident wrong prediction severely — the penalty heads to infinity as you predict 0.99 for a true 0. MSE through a sigmoid is non-convex, so log-loss is preferred.
For each example: −[y·log(p) + (1−y)·log(1−p)]. If the true label is 1, you pay −log(p) — near zero when p is high, exploding when p is low. It rewards being calibrated, not just correct.
- “How do you regularize it?” — Add an L1 or L2 penalty, exactly as in linear regression.
- “Does it handle imbalanced data?” — Use class weights or resampling, and judge with PR-AUC.
What does the decision boundary look like?
Where the probability equals 0.5, the linear score z = 0 — so the boundary is a straight line (a hyperplane in higher dimensions). Logistic regression is a linear classifier; curved boundaries need engineered features.
The boundary is linear. Points are pushed toward probability 0 or 1 by their distance from it — far points are confident, near points are uncertain.
- “Is it a linear model?” — Yes — linear in the features through the link; the boundary is a hyperplane.
- “How do you pick the threshold?” — Tune it on the precision/recall tradeoff, not always 0.5.
How does it extend to multiclass?
Replace the sigmoid with the softmax, which turns a vector of class scores into probabilities that sum to 1. Train it with categorical cross-entropy. (Or use one-vs-rest: a separate binary classifier per class.)
eᶻⁱ / Σ eᶻ — exponentiate each score and normalize. The largest score wins, but you keep a full distribution.
Train one classifier per class against all others; pick the highest score. Simple but probabilities aren't normalized.
The multiclass generalization of log-loss — penalizes low probability on the true class.