Linear Regression
What is linear regression and what does it assume?
Linear regression fits a straight line (or hyperplane) ŷ = w·x + b that predicts a continuous target from features. It assumes a roughly linear relationship, independent observations, constant error spread (homoscedasticity), and roughly normal residuals.
The target is a weighted sum of the features. Curved patterns need feature transforms.
Residual variance is constant across the range of predictions.
Errors are roughly normal — needed for valid confidence intervals and p-values.
- “How do you model non-linearity?” — Add polynomial or interaction features; it's still linear in the parameters.
- “What if assumptions are violated?” — Transform features/target, add regularization, or switch to a non-linear model.
- “Correlation vs regression?” — Correlation measures strength of association; regression models the actual functional relationship.
What's the cost function, and how is the best-fit line found?
It minimizes the Mean Squared Error — the average squared vertical distance from each point to the line. Squaring punishes big misses and makes the problem smooth and convex, so there's a single best line.
- “Why squared error, not absolute?” — Squaring is smooth and differentiable everywhere with a closed-form solution; absolute error (MAE) is more robust to outliers but harder to optimize.
OLS vs gradient descent — how is it actually solved?
Two routes to the same minimum: the closed-form normal equation (OLS) solves it in one shot, while gradient descent iterates toward it — the better choice when you have many features or rows.
- Exact, one-step:
w = (XᵀX)⁻¹Xᵀy. - No learning rate to tune.
- Slow / unstable when features are many (matrix inverse is ~O(d³) in the number of features).
- Iterative; scales to huge / high-dimensional data.
- Works online and for models with no closed form.
- Needs a learning rate and feature scaling. (See Gradient Descent.)
How do you measure how good the fit is (R²)?
R² is the fraction of the target's variance the model explains: R² = 1 − SS_res / SS_tot. R² = 1 is perfect, 0 is no better than predicting the mean, and it can go negative for a truly bad model.
Plain R² never decreases when you add features, even useless ones. Use adjusted R², which penalizes extra features, to compare models with different feature counts. Pair it with RMSE / MAE for error in real units.
What is multicollinearity and why does it matter?
Multicollinearity is when two or more features are highly correlated. The model can't tell which one drives the target, so the coefficients become unstable and hard to interpret — though predictions can still be fine.
Tiny data changes swing coefficients wildly, sometimes flipping their sign.
The Variance Inflation Factor flags features that are well-predicted by the others.
Remove a redundant feature, combine them, or use L2 (Ridge), which stabilizes correlated weights.