"Should we fine-tune a model?" is one of the most common questions we get from teams starting an AI initiative — and it's usually the wrong first question. The right first question is: what kind of knowledge gap are you actually trying to close?
Two different problems
Retrieval-augmented generation (RAG) gives a model access to information it wasn't trained on, at the moment it needs it — your internal docs, a customer's account history, a product catalog that changes daily. The model's underlying behavior doesn't change; what changes is what it can see.
Fine-tuning changes how the model behaves — its tone, its output format, its judgment on ambiguous cases within a narrow domain. The knowledge the model already has doesn't change much; what changes is how it uses that knowledge.
When RAG is the right call
- The information changes frequently (pricing, inventory, policy documents).
- You need to cite a source for every answer — RAG makes that traceable, fine-tuning doesn't.
- The knowledge is proprietary and you don't want it baked into model weights you might switch providers on later.
When fine-tuning earns its cost
- You need consistent structured output at high volume, and prompting alone is inconsistent.
- The task requires a narrow judgment call repeated thousands of times — classification, extraction, tone-matching — where examples teach the pattern better than instructions do.
- Latency or cost matters enough that a smaller fine-tuned model can replace a larger general one for a specific task.
Why most enterprise systems end up using both
A support assistant might use RAG to pull the right policy document and a lightly fine-tuned model to keep its tone consistent with brand voice across thousands of interactions. Neither technique alone solves both problems. The mistake is treating this as an either/or architecture decision made once, instead of two separate levers you tune independently as the system matures.
Start with RAG. It's cheaper to change, easier to audit, and solves the knowledge-freshness problem that breaks most AI pilots in the first place. Reach for fine-tuning once you know exactly what behavior you're trying to lock in.
The takeaway
Don't ask "RAG or fine-tuning" as an abstract architecture question. Ask what's actually failing today — stale or missing knowledge, or inconsistent behavior — and let that answer choose the technique.