Support Vector Machines
What is the maximum margin and why does it matter?
An SVM finds the separating line that sits as far as possible from both classes — the widest “street.” Only the closest points, the support vectors, define it. A wide margin generalizes better and is robust to small perturbations.
The boundary is placed to maximize the gap. Only the support vectors (circled) matter — move any other point and the boundary doesn't change.
- “Why scale features for SVM?” — Distances drive everything; unscaled features distort the margin.
- “SVM vs logistic regression?” — SVM maximizes the margin (hinge loss); logistic regression models probabilities (log-loss). SVM shines in high dimensions; LR gives calibrated probabilities.
What is the kernel trick?
When data isn't linearly separable, the kernel implicitly maps it into a higher-dimensional space where a flat boundary does separate it — computing the needed dot products without ever building those high-dimensional coordinates.
No mapping — for already-separable or high-dimensional data like text.
Adds feature interactions up to a chosen degree.
Infinite-dimensional; carves flexible, local boundaries. Tune γ for reach.
The optimization only needs dot products between points. A kernel returns that dot product in the mapped space directly — so you get the power of a huge feature space at the cost of the original one.
- “Does it scale to big data?” — Kernel SVMs are roughly O(n²)–O(n³); for millions of rows prefer linear SVM or boosting.
What does the soft-margin C parameter do?
Real data overlaps, so SVMs allow some points to sit inside or across the margin. C sets how much you punish those violations — it's the bias-variance knob for SVMs.
- Tolerates more violations.
- Smoother boundary, higher bias, lower variance.
- Good when data is noisy.
- Punishes every violation.
- Tighter fit, lower bias, higher variance.
- Risks overfitting noisy data.
- “What's the loss function?” — Hinge loss plus an L2 penalty on the weights (the margin maximization).
SVM vs logistic regression — when to use which?
Both are linear classifiers, but they optimize different objectives: SVM maximizes the margin (hinge loss), logistic regression maximizes likelihood (log loss) and outputs calibrated probabilities.
- Strong with clear margins and high-dimensional data; kernels for non-linearity.
- No native probabilities; slower on huge datasets.
- Fast, probabilistic, interpretable coefficients.
- Linear boundary unless you engineer features. See logistic regression.
How does SVM handle multiclass and probabilities?
SVM is binary at heart. Multiclass is built from many binary SVMs via one-vs-rest or one-vs-one voting. Probabilities aren't native — they're estimated post-hoc (Platt scaling).
- “What are support vectors again?” — The training points on or inside the margin; only they define the boundary, which is what makes SVM memory-efficient at prediction time.