NER & POS Tagging
What is POS tagging?
Part-of-speech tagging labels each word with its grammatical role — noun, verb, adjective, etc. It's a sequence-labeling task (one tag per token) and a building block for parsing and information extraction.
One label per token. The same word can take different tags in different sentences — tagging is context-dependent.
- “Why is POS context-dependent?” — “book” is a noun in “a book” but a verb in “book a flight.”
What is Named Entity Recognition?
NER finds and classifies real-world entities — people, organizations, locations, dates, amounts. It powers search, knowledge graphs, resume parsing, and redaction.
NER labels spans (which can be multiple tokens, like “Tim Cook”) with an entity type — typically using the BIO tagging scheme.
- “What is BIO tagging?” — B-(egin), I-(nside), O-(utside) tags mark entity spans so multi-word entities are captured (B-PER, I-PER).
How are these modeled?
The field evolved from hand-written rules → statistical sequence models (HMM, CRF) → BiLSTM-CRF → fine-tuned transformers (BERT), which are now state of the art.
What makes them hard?
Ambiguity and context. “Apple” is a company or a fruit; “Washington” is a person, a state, or a city — only context decides. New and rare entities, and nested spans, add difficulty.
- “How is NER evaluated?” — Span-level precision, recall, and F1 — an entity counts only if its boundaries and type both match.
- “What is entity linking?” — Mapping a recognized mention to a unique entry in a knowledge base (which “Washington”?).