Sampling & Decoding
How does the model pick the next token?
At each step the model outputs a probability distribution over the whole vocabulary (logits → softmax). Decoding is the strategy for choosing one token from that distribution — always the top one (greedy) or a sample.
- “What are logits?” — The raw pre-softmax scores; softmax turns them into probabilities that sum to 1.
What does temperature do?
Temperature divides the logits before softmax. Low T sharpens the distribution (the top token dominates → focused, deterministic). High T flattens it (more options get a chance → diverse, creative, riskier).
- “What is T = 0?” — Effectively greedy decoding — always the most likely token, fully deterministic.
Greedy vs top-k vs top-p (nucleus)?
They differ in which tokens are eligible to sample from — i.e. how they truncate the distribution before sampling.
Greedy takes the single best token; beam search keeps several candidate sequences. Safe but repetitive.
Sample only from the k highest-probability tokens — cuts off the long tail of junk.
Keep the smallest set of tokens whose probability sums to p (e.g. 0.9) — adapts to how peaked the distribution is.
- “Top-k vs top-p?” — Top-p is usually preferred: it widens when the model is unsure and narrows when it's confident.
When do you use low vs high temperature?
Low for correctness, high for creativity.
- Facts, code, extraction, classification.
- Reproducible, fewer hallucinations.
- Brainstorming, story writing, variety.
- More surprising — and more likely to drift.
- “Lower temperature to reduce hallucinations?” — It helps a bit, but grounding/RAG matters far more. (See Hallucinations.)
- “Why isn't output reproducible even at T=0?” — Hardware/floating-point non-determinism and batching can still cause small variations.