LLMs & How They Work
What is an LLM and how does it generate text?
An LLM is a giant transformer trained to predict the next token. It generates autoregressively: predict one token, append it, feed the whole thing back in, predict the next — repeating until done.
- “Is it just memorizing?” — No — it learns statistical patterns of language; it generalizes, but can also reproduce memorized text.
- “How does it pick each token?” — From a probability distribution over the vocabulary. (See Sampling & Decoding.)
How is an LLM trained?
Three stages: pretraining on a huge text corpus to predict next tokens (the bulk of the cost), instruction tuning on example tasks to follow directions, and RLHF to align outputs with human preferences.
Self-supervised on trillions of tokens — learns grammar, facts, reasoning patterns.
Supervised fine-tuning on (instruction, good answer) pairs so it becomes helpful.
A reward model trained on human preferences steers it toward helpful, harmless responses.
- “What's self-supervised learning?” — Labels come free from the data itself (the next word), so no manual annotation is needed.
- “Pretraining vs fine-tuning?” — Pretraining builds general ability; fine-tuning specializes it. (See Fine-tuning vs Prompting.)
What are tokens and the context window?
LLMs read tokens — sub-word chunks (~¾ of a word on average), not characters or whole words. The context window is the max tokens the model can attend to at once; the prompt and the response share that one budget.
Cost and latency scale with tokens, and anything beyond the window is invisible to the model. A long chat must be summarized or truncated to fit. See Context Windows.
- “Why sub-word tokens?” — They balance vocabulary size against handling rare/unknown words. (See Tokenization.)
- “What's 'lost in the middle'?” — Models attend best to the start and end of a long context, weakest in the middle.
What are the main limitations?
LLMs can hallucinate (state false things confidently), have a fixed knowledge cutoff, no built-in memory between calls, and they reflect biases in their training data.
- “How do you give it fresh / private knowledge?” — Retrieval-augmented generation. (See RAG.)
- “How do you reduce hallucinations?” — Grounding, retrieval, lower temperature, asking it to cite. (See Hallucinations.)
- “What are emergent abilities?” — Capabilities (multi-step reasoning, in-context learning) that appear only past a certain scale.