RAG — Retrieval-Augmented Generation
What is RAG and why use it?
RAG retrieves relevant documents and pastes them into the prompt so the LLM answers from your data, not just its training. It fixes the big gaps: stale knowledge, no private data, and hallucinations — without retraining the model.
Update the document store and the model instantly “knows” new facts.
Answers are anchored to retrieved text you can cite and verify.
Answer over internal wikis, PDFs, tickets — data the base model never saw.
- “Why not just put everything in the prompt?” — The context window is limited and costly; retrieval sends only the relevant chunks.
How does the RAG pipeline work?
Offline, documents are chunked and embedded into a vector database. At query time: embed the question, retrieve the nearest chunks, stuff them into the prompt, and let the LLM generate a grounded answer.
- “What is chunking?” — Splitting documents into passages so retrieval returns focused, relevant pieces (with some overlap).
- “What's a reranker?” — A second model that re-scores retrieved chunks for relevance before they go to the LLM.
RAG vs fine-tuning — which when?
RAG injects knowledge at query time; fine-tuning bakes in behavior/style. For facts that change or are private, use RAG. For a consistent format, tone, or skill, fine-tune. They combine well.
- Knowledge that updates often or is proprietary.
- You need citations / verifiability.
- A fixed output style, format, or domain skill.
- Latency-sensitive (no retrieval step). See Fine-tuning.
- “Can you use both?” — Yes — fine-tune for behavior, RAG for facts. Very common in production.
Why does RAG fail, and how do you improve it?
Most RAG failures are retrieval failures: if the right chunk isn't fetched, the LLM can't use it. Garbage in, garbage out.
Better chunking, a stronger embedding model, hybrid search (keyword + vector), a reranker, more retrieved chunks, query rewriting, and prompting the model to say “I don't know” when context is missing.
- “How do you evaluate RAG?” — Measure retrieval (recall@k) and generation (faithfulness/groundedness, answer relevance) separately.
- “What is hybrid search?” — Combining sparse keyword (BM25) with dense vector search to catch both exact terms and semantic matches.