Naive Bayes
What is Bayes' theorem, in plain terms?
It updates a belief with evidence: posterior ∝ prior × likelihood. To classify, compute that for every class and pick the largest — you don't even need the denominator, since it's the same for all classes.
Pick the class that maximizes prior × likelihood. The prior is how common the class is; the likelihood is how well the class explains the observed features.
- “Generative or discriminative?” — Generative: it models P(features | class); logistic regression is the discriminative counterpart.
What's the "naive" assumption, and why make it?
It assumes every feature is conditionally independent given the class. That's almost never literally true, but it collapses one impossible joint probability into a simple product of per-feature probabilities — which is what makes it fast and data-efficient.
Estimating the full joint P(x₁,…,xₙ | class) needs exponentially much data.
∏ P(xᵢ | class) — estimate each feature on its own. Trains in one pass.
Add a small count so an unseen feature doesn't zero out the whole product.
- “Why work in log space?” — Summing log-probabilities avoids floating-point underflow from multiplying many small numbers.
Why does it work so well for text even when the assumption is wrong?
For classification you only need the right class to score highest, not accurate probabilities. Even with correlated words, the ranking usually comes out right — so spam filters and topic classifiers built on Naive Bayes are fast, strong baselines.
- Extremely fast to train and predict.
- Works with little data and very high dimensions (bag-of-words).
- A strong, hard-to-beat baseline for text.
- Probabilities are poorly calibrated (often over-confident).
- Correlated features get double-counted.
- Can't learn feature interactions.
- “Which variant?” — Multinomial for word counts, Bernoulli for binary presence, Gaussian for continuous features.
- “When does it lose?” — When feature interactions matter and you have plenty of data — then trees or logistic regression win.
How does it handle continuous features?
Gaussian Naive Bayes assumes each numeric feature is normally distributed per class — it estimates a mean and variance per feature per class and plugs them into the Gaussian density to get the likelihood.
- “What if a feature isn't Gaussian?” — Transform it (e.g. log), or discretize into bins and use the multinomial form.
Naive Bayes vs logistic regression — when does each win?
A classic generative/discriminative pair. Naive Bayes (generative) needs less data and trains instantly; logistic regression (discriminative) makes no independence assumption and usually wins with enough data.
- “Small-data behavior?” — Naive Bayes often beats logistic regression when data is scarce, then logistic regression overtakes it as data grows.