Feature Scaling
Why does feature scaling matter?
Many algorithms compare or combine features by their raw magnitude. If income is in the thousands and age is in the tens, the larger-range feature dominates distances and gradients — not because it's more important, but because its numbers are bigger.
Euclidean distance is swamped by the widest-range feature unless scaled.
Scaling makes the loss surface rounder so gradient descent converges quicker.
L1/L2 penalize coefficients equally, so features must be on a comparable scale.
Standardization vs normalization — what's the difference?
Both rescale, but to different shapes. Standardization centers to mean 0, std 1; min-max normalization squeezes into a fixed range like [0, 1].
z = (x − μ) / σ- No fixed bounds; preserves outlier shape.
- Default for most models; assumes roughly Gaussian-ish data.
x' = (x − min) / (max − min)- Bounded to [0, 1]; good for images and neural nets.
- Very sensitive to outliers — one extreme value squashes the rest.
- “What about RobustScaler?” — It centers on the median and scales by the IQR, so outliers barely move it.
Which algorithms need scaling and which don't?
Rule of thumb: anything based on distance, dot products, or gradient descent needs it; tree-based models don't.
- KNN, K-Means, SVM (RBF), PCA
- Linear / logistic regression with regularization
- Neural networks
- Decision trees, Random Forest, gradient boosting
- They split on thresholds per feature, so monotonic rescaling doesn't change splits.
How do you scale without leaking data?
The scaler's statistics (mean, std, min, max) must come from training data only. Fit on train, then apply the same transform to validation and test.
Split → fit the scaler on train → transform train, val, and test with those same parameters. Wrapping it in a pipeline and using cross-validation does this automatically per fold. Fitting on the full dataset first is a textbook leakage bug.
How do outliers affect scaling?
Outliers distort the statistics scaling depends on. A single huge value blows up the max (wrecking min-max) and inflates the std (shrinking everyone else under standardization).
- “What would you use with heavy outliers?” — RobustScaler (median + IQR), or cap/winsorize and log-transform first.
- “Should you scale the target?” — Sometimes for regression (log or standardize a skewed target), but remember to invert the transform on predictions.