Tokenization
What is tokenization, and why sub-word?
Tokenization splits text into the units a model reads. Modern LLMs use sub-word tokens — a sweet spot between whole words (huge vocab, breaks on unknown words) and characters (tiny vocab, very long sequences).
- “Roughly how many tokens per word?” — About 1.3 on average for English — common words are one token, rare ones split into pieces.
How does Byte-Pair Encoding work?
BPE starts from individual characters and repeatedly merges the most frequent adjacent pair into a new token, until it reaches the target vocabulary size. Common chunks (“ing”, “low”) become single tokens; rare words stay split.
- “BPE vs WordPiece vs SentencePiece?” — Variants of the same idea: WordPiece (BERT) merges by likelihood; SentencePiece works on raw text with no pre-splitting.
Word vs character vs sub-word?
It's a trade-off between vocabulary size and sequence length.
Short sequences but a massive vocabulary, and any unseen word is “unknown.”
No unknown-word problem, but sequences get very long and meaning is harder to learn.
Moderate vocab, reasonable length, and rare words decompose into known pieces. The default.
- “How does sub-word handle unknown words?” — It splits them into known sub-word pieces, so there's no hard out-of-vocabulary failure.
Why does tokenization matter for LLMs?
Tokens are the unit of cost, context, and speed: pricing and the context window are counted in tokens, and odd tokenization causes quirks — like LLMs struggling to count letters or do digit math.
- “Why are some languages more expensive?” — Tokenizers trained mostly on English split other scripts into more tokens per word.
- “Why can't it spell / count characters well?” — It sees tokens, not individual letters, so character-level tasks are awkward.
- “Connection to context windows?” — The window is a token budget shared by prompt and response. (See LLMs.)