Text Preprocessing
What's in a typical preprocessing pipeline?
For classic NLP you clean and normalize text into consistent tokens: lowercase → strip noise → tokenize → remove stop words → stem/lemmatize. The goal is to collapse surface variation so the same concept maps to the same feature.
A classic cleaning pipeline. Each step reduces noise so “Running,” “runs,” and “ran” can collapse toward one feature.
- “What is normalization?” — Standardizing case, punctuation, accents, contractions, and numbers so equivalent text matches.
Stemming vs lemmatization?
Both reduce words to a base form. Stemming chops affixes with crude rules (fast, can produce non-words); lemmatization maps to the real dictionary word using vocabulary and part-of-speech (slower, accurate).
- “studies” → “studi”, “running” → “run”.
- Rule-based, fast, can output non-words.
- “studies” → “study”, “better” → “good”.
- Dictionary + POS aware; real words, slower.
- “When use which?” — Stemming for speed at scale (search indexing); lemmatization when correctness/readability matters.
Should you remove stop words?
Stop words (“the”, “is”, “of”) are high-frequency, low-information words. Removing them helps count-based models — but it can hurt when those words carry meaning (“to be or not to be”, “the” in “The Who”).
- “When is it risky?” — Sentiment (“not good”) and negation depend on stop words — don't strip them blindly.
Do transformer models still need this?
Mostly no. BERT/GPT take raw text through their own sub-word tokenizer and learn from context — so stemming, stop-word removal, and lowercasing are usually skipped (and can even remove useful signal).