Feature Engineering
What is feature engineering and why does it still matter?
Feature engineering is turning raw data into inputs a model can learn from — creating, transforming, encoding, and selecting variables. For classical models on tabular data it's often where most of the accuracy comes from: a good feature beats a fancier algorithm.
Logs for skewed values, scaling, binning a continuous variable into ranges.
Ratios, differences, interaction terms, date parts (day-of-week, hour).
One-hot, ordinal, or target encoding so the model can use them.
- “Does deep learning remove the need for it?” — For images, text, and audio, networks learn features themselves. For tabular data, hand-built features still dominate.
- “Where do good features come from?” — Domain knowledge and error analysis, not just automated transforms.
How do you handle categorical features?
Pick the encoding by whether the categories are ordered and how many there are. See encoding categorical variables for the full picture.
- One-hot — one binary column per category; no false ordering.
- Ordinal — integer codes only when categories have a real order (low/med/high).
- Target / mean encoding — replace with the mean target, but fit inside CV folds to avoid leakage.
- Hashing / embeddings — for huge spaces like user or product IDs.
- “Why not just label-encode everything?” — Integer codes imply an order and distance that nominal categories don't have, which misleads linear models and distance-based ones.
- “How handle a category unseen at train time?” — Reserve an “unknown” bucket or fall back to the global mean.
How do you handle missing values and outliers?
Missing data is rarely random — the fact that a value is missing can itself be predictive, so don't just drop it blindly.
Median for skewed numerics; a constant or most-frequent for categoricals. Fit the imputer on train only.
Add an “is_missing” column so the model can learn from the absence itself.
Winsorize or log-transform; tree models tolerate outliers, linear models don't.
- “Drop rows or columns with missing data?” — Drop only if the column is mostly empty or the rows are a tiny, random fraction.
How do you select which features to keep?
More features isn't better — irrelevant ones add noise, cost, and overfitting risk. Three families of methods:
Correlation, mutual information, variance thresholds. Fast first pass.
Recursive feature elimination, forward/backward selection. Accurate but expensive.
- “Two highly correlated features — keep both?” — Usually drop one; they're redundant and cause multicollinearity for linear models.
- “Are tree feature importances reliable?” — They're biased toward high-cardinality features; prefer permutation importance or SHAP for trustworthy ranking.
What is feature leakage and how do you avoid it?
Leakage is when a feature contains information that wouldn't be available at prediction time — so the model looks great offline and fails in production.
Computing scaling/encoding statistics on the whole dataset before splitting; using a future-derived column (e.g. “total spend” to predict churn); or target leakage where a feature is a proxy for the label. Fix: split first, then fit all transforms inside a pipeline on the training fold only. See cross-validation & data splits.
- “How would you catch leakage?” — A feature with suspiciously high importance or a model that's “too good” is a red flag; audit how each feature is computed relative to the prediction time.