Text Representation
Why turn text into numbers, and how?
Models do math, not words — so text must become vectors. The classic approach builds a document–term matrix: rows are documents, columns are vocabulary words, cells are counts (Bag of Words).
Bag of Words ignores order and grammar — it just counts which words appear. Each row is now a numeric vector a model can use.
- “What is one-hot encoding?” — A single word as a vector that's 1 in its slot and 0 elsewhere — sparse and with no notion of similarity.
Bag of Words vs TF-IDF?
Raw counts over-weight common words. TF-IDF multiplies term frequency by inverse document frequency, downweighting words that appear everywhere and boosting rare, distinctive ones.
- Plain counts; “the” dominates.
- Simple, but common words drown out signal.
tf × log(N / df)— rare words score higher.- Highlights the words that actually distinguish a document.
- “Why the log in IDF?” — It dampens the effect so very rare words don't completely dominate.
What do n-grams add?
A unigram bag loses word order. N-grams count short sequences (“not good”, “New York”) so some local context and phrasing survive — at the cost of a much bigger, sparser vocabulary.
- “Why not huge n?” — Feature count explodes and most n-grams appear once — sparse and overfit-prone.
- “Where do n-grams still shine?” — Fast, strong baselines for spam/sentiment with a linear classifier.
What are the limits of count-based features?
They're sparse, high-dimensional, ignore word order beyond n-grams, and have no sense of meaning: “car” and “automobile” are as unrelated as “car” and “banana.”
- “What solves the meaning problem?” — Dense word embeddings place similar words close together.
- “What about context?” — Contextual models (BERT) give a word different vectors depending on its sentence.