Fine-tuning vs Prompting
Prompting vs fine-tuning — what's the difference?
Prompting steers a frozen model with instructions and examples — no training. Fine-tuning updates the model's weights on your data, baking in a behavior or skill. Prompting is instant and cheap; fine-tuning is more powerful but costs data and compute.
- No training; iterate in seconds.
- Great for most tasks; limited by context window.
- Updates weights → consistent style/format, shorter prompts.
- Needs a curated dataset and compute; risk of overfitting.
- “What is few-shot / in-context learning?” — Putting examples in the prompt so the model infers the task — no weight updates. (See Prompt Engineering.)
What is LoRA / PEFT?
PEFT (parameter-efficient fine-tuning) trains only a tiny number of new parameters while freezing the giant base model. LoRA is the popular method: inject small low-rank adapter matrices and train just those.
LoRA freezes W and learns two small matrices whose product is added to it. You store only the tiny adapters per task, not a whole copy of the model.
- “Why is LoRA cheap?” — Few trainable params → less memory and compute, and adapters are a few MB instead of GBs.
- “What is QLoRA?” — LoRA on top of a quantized base model, so you can fine-tune big models on one GPU.
Fine-tune, prompt, or RAG — how do you choose?
Default to prompting, add RAG when you need external/fresh knowledge, and fine-tune only when you need a consistent behavior prompting can't reliably get.
Fast, free to iterate, covers most needs.
A fixed tone/format/skill, or to shorten long prompts at scale.
- “Rule of thumb?” — “RAG for what it should know, fine-tuning for how it should behave.”
What are the risks of fine-tuning?
It can overfit a small dataset, suffer catastrophic forgetting (losing general ability), and it freezes knowledge at training time — so facts still go stale.
- “What is catastrophic forgetting?” — Over-training on a narrow task degrades unrelated skills the base model had.
- “How much data do you need?” — Often hundreds-to-thousands of high-quality examples; quality > quantity.
- “Instruction tuning vs fine-tuning?” — Instruction tuning is fine-tuning on diverse (instruction, response) pairs to make a model follow directions.