RAG or Fine-Tuning: Which Does Your AI Need?

RAG or fine-tuning — a decision guide from the Canopus AI engineering team

The short answer

Use retrieval when the answer must come from your own content. Use fine-tuning when you need a consistent format, tone or classification decision rather than new facts. Try prompting first, because it costs nothing and is often enough. Fine-tuning does not reliably teach a model facts, which is what most teams want from it.

What each one actually changes

The distinction is not a spectrum. They solve different problems and are frequently used together.

Retrieval-augmented generation

Your documents are indexed. At question time the system finds the relevant passages and hands them to the model along with the question. The model reasons over content it has just been given rather than content it memorised. Because the source passage is known, the answer can carry a citation the reader can open, and the retrieval step can filter by the permissions the user already has.

Update a policy on Friday afternoon and the next answer reflects it. Nothing is retrained.

Fine-tuning

You take a base model and continue training it on examples of the behaviour you want. It learns shape extremely well — output format, house tone, a classification taxonomy specific to your business, adherence to a rigid response structure. It learns facts unreliably, and when it gets one wrong it does so with complete confidence and no citation.

Update a policy on Friday afternoon and nothing changes until you retrain, revalidate and redeploy.

The rule of thumb

If a correct answer changes when your documents change, you need retrieval. If a correct answer changes only when your style or taxonomy changes, fine-tuning is a candidate. Wanting the model to "know our business" is almost always a retrieval requirement wearing a fine-tuning costume.

Compared on what matters in production

Criterion	Retrieval (RAG)	Fine-tuning
What it changes	What the model can see	How the model behaves
Freshness of knowledge	Immediate — reindex and it is live	Frozen until the next training run
Citations back to source	Yes, by construction	No — nothing to cite
Per-user permission filtering	Yes, applied at retrieval	Not possible — knowledge is baked in
Consistent output format	Achievable with a schema	Its strongest use
Cost to update	Reindex the changed documents	Curate data, retrain, revalidate
Inference latency	Higher — retrieval adds a step	Lower, and often a smaller model
Explaining a wrong answer	Inspect what was retrieved	Largely opaque

Scroll the table sideways to compare both columns.

The order we try things in

Prompting and output schemas. Free, instant to change, and sufficient more often than anyone expects. Constrain the response shape and force an explicit "not found" path.
Retrieval. Add grounding in your own content, with citations and permission filtering. This is where the majority of business AI work lands.
Fine-tuning. Only once you have evidence that prompting cannot hold the format or the classification, and you have a labelled dataset worth training on.

Skipping to step three is common and costly, because the failure is delayed — the model looks impressive in a demo and drifts in production, with no citation to explain why.

The engineering that decides success either way

The model is one component. These are the parts that determine whether the system is trustworthy.

Chunking that respects meaning. Splitting a clause from the condition that qualifies it produces confidently wrong answers. Semantic chunking with overlap beats fixed-size splitting on every document set we have measured.
Hybrid search. Vector similarity plus keyword matching, then re-ranking. Pure vector search misses exact identifiers — part numbers, clause references, error codes — which is precisely what people search for.
Permission-aware retrieval. Filter before the model sees anything. A support agent and a director asking the same question should get different answers from the same index.
An evaluation set that runs in CI. Fifty to two hundred real questions with answers your experts agree on. A prompt edit that quietly breaks accuracy then fails the pipeline instead of a customer.
PII redaction before inference, and audit logging after it. Under the enterprise API terms we deploy on, API traffic is contractually excluded from training by default; where data cannot leave your infrastructure at all, open-weight models on your own instances are the route.

Where we tell clients not to use AI at all

If the rule is deterministic, write the rule. A model that gets a tax calculation right most of the time is worse than a function that gets it right every time, and it costs more to run. We have replaced proposed AI features with a lookup table more than once, and the client kept the budget.

Proving it before you commit

AI is the one service line where we insist on a short paid evaluation before quoting a build. Two weeks: week one builds the test set of real questions with agreed answers, week two builds a deliberately ugly prototype against real data. What comes back is measured accuracy, cost per call, latency, and the gap to what your process actually requires.

Roughly one project in five stops there. That is the point of it — a fortnight is a much better way to discover an accuracy ceiling than a quarter. You keep the test set, the measurements and the written recommendation either way.

What we will not do is quote an accuracy percentage before measuring on your data. Any number offered in a first meeting is a guess wearing the costume of a commitment.

The full approach is on AI and intelligent systems. Data handling, redaction and what we explicitly do not certify are covered in security and compliance. Since an assistant is only as useful as the systems it can read from and write to, API and system integration is usually the other half of the work. More guides are on the insights index.

Published: 23 August 2026 Last updated: 23 August 2026 Written by the Canopus delivery team

RAG or fine-tuning: which does your AI need?