PCA & Dimensionality Reduction
What is PCA and why use it?
PCA finds a smaller set of new axes — principal components — that capture as much of the data's variance as possible. Project onto the top few and you compress high-dimensional data while losing very little information.
Cut hundreds of correlated features down to a handful — faster training, less storage.
Project to 2 or 3 components to actually see high-dimensional structure.
Low-variance components are often noise; dropping them can improve downstream models.
- “Is PCA supervised?” — No — it ignores labels. LDA is the supervised cousin that maximizes class separation.
How does it work?
It finds the direction of maximum variance (PC1), then the next orthogonal direction (PC2), and so on — these are the eigenvectors of the data's covariance matrix. Then it projects every point onto those axes.
PC1 (green) runs along the cloud's longest spread; PC2 (orange) is perpendicular. Projecting each blue point onto PC1 keeps most of the variation in a single new feature.
- “Why standardize first?” — PCA chases variance, so an unscaled large-range feature would dominate every component.
- “What are the components, mathematically?” — Eigenvectors of the covariance matrix; eigenvalues are the variance each explains (often via SVD).
How many components do you keep?
Each component has an explained-variance ratio. Keep enough top components to reach a target — say 95% cumulative variance — read straight off the scree plot.
The first few components carry most of the variance; the tail is near-noise. Keep components up to your cumulative-variance threshold and drop the rest.
- “Downsides?” — Components are linear combinations, so interpretability is lost; it can't capture non-linear structure (use t-SNE / UMAP for that).
Why must you scale before PCA?
PCA finds directions of maximum variance, and variance depends on units. Without standardization, a feature measured in large numbers (e.g. salary) dominates the components purely because of its scale, not its importance.
- “Center too?” — Yes — PCA assumes mean-centered data; libraries usually do this for you.
What are PCA's limitations?
PCA is linear and unsupervised, so it can miss what matters.
Max-variance directions aren't always the most discriminative — LDA is the supervised counterpart.
Components are combinations of all features, so they're harder to explain.