RAG or fine-tuning? A decision framework

This question comes up in nearly every AI engagement, and it is usually framed as a choice. It is not really a choice — the two techniques solve different problems, and most production systems end up using both. The useful question is which one solves your problem, and in what order.

What each one actually does

Retrieval-augmented generation puts relevant documents into the model's context at query time. The model does not learn anything permanently; it reads what you hand it and answers from that. Update the documents and the answers update immediately.

Fine-tuning adjusts the model's weights on examples of the behaviour you want. It does not teach facts reliably — it teaches shape: how to format output, what tone to hold, how to perform a specific task.

The most common expensive mistake is fine-tuning to teach facts. It appears to work in testing, then fails unpredictably, and every fact update requires retraining.

Choose RAG when

• The knowledge changes — pricing, policies, documentation, inventory, case law
• Answers must cite a source, which regulated and public sector work almost always requires
• Access control matters, because retrieval can respect per-document permissions and weights cannot
• You need to add or remove information immediately
• You cannot risk the model asserting something confidently that is no longer true

Choose fine-tuning when

• Output must follow a strict format the base model keeps drifting from
• You need a consistent voice across thousands of generations
• The task is specialised enough that prompting is unreliable — domain classification, structured extraction from unusual documents
• Latency or cost matters and a smaller fine-tuned model can replace a larger prompted one
• You have several hundred high-quality examples, which is the part people underestimate

Use both when

The common production shape is a fine-tuned model that has learned how to answer, with retrieval supplying what to answer from. A support assistant might be fine-tuned on your tone and escalation rules, while retrieving live policy documents for every response.

Build the evaluation first

Whichever you pick, build the test before the system. A task set with graded answers turns "does this seem better?" into a number.

Without it you cannot tell whether a prompt change helped, whether a model upgrade regressed something, or whether retrieval is returning the right documents. Teams that skip this end up tuning by vibes and discovering regressions through customer complaints.

We build the evaluation harness in the first week of an AI engagement, before any retrieval or tuning work. It is the single highest-leverage thing in an AI project, and it is the thing most often left out.

What people get wrong

Fine-tuning to inject knowledge. As above — it partially works, which is worse than not working, because it fails silently.

Retrieving too much. More context is not better context. Precision matters more than recall past a fairly low threshold, and stuffing the window degrades answers while raising cost.

Ignoring chunking. How documents are split determines what can be retrieved. Splitting mid-table or mid-clause makes the content unusable, and no amount of model quality recovers it.

No freshness strategy. A retrieval index that is not updated becomes confidently wrong, which is worse than empty.

A reasonable default

Start with retrieval and a strong prompt. Measure it. Fine-tune only where the evaluation shows a specific, repeatable failure that retrieval and prompting cannot fix.

That order is cheaper, faster to change, and easier to explain to whoever eventually asks why the system said what it said.

---

We build AI systems with the evaluation, retrieval and guardrails designed in from the start. See how we approach AI systems engineering, or tell us what you are trying to build.