Insight · ·

RAG vs fine-tuning: diagnose knowledge and behaviour separately

Use retrieval for changing source material and fine-tuning for a demonstrated behavioural gap. Neither replaces evaluation or permissions.

RAG vs fine-tuning: diagnose knowledge and behaviour separately

Name the failure before choosing the technique

If an assistant cannot find the current policy, start by examining retrieval. If it has the correct evidence but repeatedly performs the task in the wrong way, investigate prompting, examples and possibly fine-tuning. Choosing a technique before classifying the failure can make the system more expensive without making it more useful.

Microsoft's guidance makes the same high-level distinction: RAG grounds answers in private or changing data, while fine-tuning targets behaviour, style or task performance. This is a decision aid, not a guarantee that either approach will succeed on a particular dataset. Microsoft RAG guidance

The useful comparison is therefore not which technique is more advanced. It is which part of the observed error each intervention can plausibly address.

Separate missing evidence from mishandled evidence

Take an illustrative support assistant answering a warranty question. The relevant policy may be absent from the index, present but not retrieved, retrieved without the necessary exception, or included correctly and then misread.

Those failures require different experiments. Adding training examples does not establish that tomorrow's policy will be retrieved. Increasing retrieval volume does not establish that the model can interpret a condition correctly. A single answer-quality score can hide this distinction.

Record the question, the authoritative source, the retrieved material and the final answer. Ask reviewers to identify where the process first diverged from the expected result. This gives the team a repair target rather than another vague request to improve the model.

Retrieval brings a source lifecycle

RAG needs a decision about which sources are authoritative, how updates arrive and what happens when a document is withdrawn. It also needs permissions applied before restricted material is exposed to the model or user.

For a policy assistant, prefer an answer that points to the relevant passage and admits when the available material is insufficient. A plausible answer without supporting evidence can be less useful than a clear refusal to resolve the question.

Changing a document should trigger a test of the resulting answer, not merely a check that indexing completed. Our guide to useful RAG products covers the broader retrieval, permission and freshness responsibilities.

Fine-tuning needs a stable target and clean examples

Consider fine-tuning when the desired behaviour can be described consistently, prompting has been evaluated, and representative examples exist. Potential targets include a specialised output convention or a recurring domain-specific task. Suitability still depends on the selected model and training method.

Keep a separate evaluation set that was not used to construct the training examples. Otherwise the experiment can reward memorisation of familiar cases rather than useful performance on new work.

Training data should represent approved behaviour, not every historical response. Old tickets may contain contradictions, sensitive information or workarounds nobody should repeat. A cleaned dataset has an owner and a documented purpose.

Retrieval and fine-tuning can also be combined. A tuned model may consume retrieved evidence, but this does not remove the need to test citations, permissions and unsupported questions.

Run a targeted comparison this week

Collect representative failures and label them as source absence, retrieval failure, reasoning error, output mismatch or policy violation. Repair the simplest demonstrated gap first.

Compare the existing system with a bounded retrieval change or a bounded behaviour change, holding the test questions stable. Include updated source material, unseen cases and requests the assistant should decline.

Clodron's AI assistant engineering should make this diagnosis measurable. The decision is justified when accepted outcomes improve under the same conditions, not when the architecture diagram gains another model-training box.