Context Windows
What is a context window?
The context window is the maximum number of tokens a model can attend to at once — everything it “sees”: the system prompt, conversation history, retrieved documents, and the response being generated. Run past it and the earliest tokens fall out of view.
- “Tokens vs words?” — A token is a sub-word chunk; roughly ¾ of a word in English. See tokenization.
- “Does output count toward it?” — Yes — input + generated output share the same budget.
Why is a bigger context window expensive?
Standard self-attention compares every token with every other token, so cost and memory scale quadratically — O(n²) — with sequence length. Doubling the context roughly quadruples the attention work.
Efficient attention (FlashAttention), sparse/sliding-window attention, and better positional schemes (RoPE, ALiBi) reduce the cost or let models extrapolate beyond their training length. A KV cache avoids recomputing past tokens during generation.
What happens when you exceed it?
The model simply can't see overflowing tokens. You must manage the budget so important content stays in view.
Cut the earliest history — risks losing important context.
Replace old turns with a running summary.
Pull only relevant chunks in on demand instead of stuffing everything.
What is the lost-in-the-middle problem?
Even within the window, models attend unevenly — they recall information at the start and end of a long context far better than content buried in the middle. A bigger window doesn't guarantee the model uses all of it well.
- “Practical takeaway?” — Put the most important instructions or evidence near the beginning or end, not the middle.
Long context vs RAG — does more context kill RAG?
No. Even with huge windows, RAG is usually cheaper, faster, and more accurate than dumping everything into context — it sends fewer tokens, avoids lost-in-the-middle, and keeps knowledge fresh and updatable.
- “So when is long context better?” — When reasoning needs the whole document at once (e.g. analyzing one long contract) rather than a few retrievable snippets.