Teams often ask us to “fine-tune a model” when what they actually need is retrieval. The two are not interchangeable — they solve different problems, cost very different amounts, and fail in different ways. Here is the decision guide we walk clients through.
What each one does
RAG (retrieval-augmented generation) gives the model your information at the moment it answers. When a question comes in, the system searches your documents, hands the most relevant passages to the model, and the model writes a grounded response — ideally citing its sources. The model itself never changes.
Fine-tuning changes the model itself by training it further on your examples. It is how you teach a consistent tone, a strict output format, or a narrow, repeatable skill. What it does not do reliably is teach the model new facts — fine-tuned models still invent things; they just invent them in your house style.
That last sentence surprises people, so it bears repeating: fine-tuning is not how you get a model to know your business. Retrieval is.
The decision matrix
| You need… | Use |
|---|---|
| Answers grounded in your documents, policies or product data | RAG |
| Information that changes weekly (prices, inventory, docs) | RAG |
| Sources cited so users can verify answers | RAG |
| A consistent tone, persona or brand voice | Fine-tuning |
| Strict output structure (always valid JSON, a fixed report format) | Fine-tuning |
| A narrow classification or extraction task at high volume | Fine-tuning (often a small model) |
| Lower latency / cost than a huge prompt allows | Fine-tuning |
| Domain knowledge and consistent behaviour | Both — RAG for facts, a light fine-tune for form |
What they cost to run
RAG costs show up in engineering: a document pipeline, an index, retrieval quality work (chunking, ranking, filtering), and evaluation. Once built, updating knowledge is trivial — change the documents and the next answer reflects them. No retraining, ever.
Fine-tuning costs show up in data and repetition: hundreds to thousands of good training examples (usually the expensive part — someone has to write or curate them), a training run, evaluation against a baseline — and then the whole loop again every time your needs shift or a better base model ships. That last point stings in practice: a fine-tune is married to its base model. When a clearly better model arrives — which in this field is roughly every quarter — your RAG system upgrades by changing one line; your fine-tune goes back to training.
Where each one goes wrong
The classic RAG failure is retrieval, not generation: the model answers beautifully from the wrong passage. The fix is unglamorous search engineering — better chunking, better ranking, filters by recency or source — which is why “we added a vector database” is the beginning of a RAG build, not the end.
The classic fine-tuning failure is expecting knowledge: teams train on 500 support tickets and are baffled that the model still misquotes the refund policy. It learned the shape of support answers, not the facts. The second classic failure is overfitting — a model so tuned to old examples it handles novel cases worse than the base model did.
The hybrid pattern (very common in production)
Mature systems frequently use both, each doing the job it is good at:
- RAG supplies the facts — current, cited, updateable.
- A light fine-tune (or a small fine-tuned model) supplies the behaviour — the tone, the format, the routing decision, the extraction step.
A support assistant might use a fine-tuned small model to classify and route the ticket, then RAG over your help centre to draft the grounded reply. Cheap where it can be, accurate where it must be.
Why we usually start with RAG
It is faster to ship, cheaper to maintain, easier to debug (you can see what was retrieved), and it upgrades for free as base models improve. Most “we need a custom model” requests we hear are solved better and cheaper with good retrieval — and when a fine-tune genuinely earns its keep, the RAG system you built first provides the evaluation data to prove it.
That is the pattern across most of the LLM and generative AI apps we build: retrieval first, fine-tuning where measurement says it pays. If budget is the constraint, our cost guide covers what each path typically runs.
Not sure which fits your case? Ask us — we will point you to the cheaper path that works, and tell you plainly if you need neither.