This question gets asked backwards. Teams ask "should we fine-tune?" when the real question is "what is the model getting wrong?" The answer to the second determines the answer to the first.

Here's the distinction that resolves most of the confusion: retrieval changes what the model knows. Fine-tuning changes how it behaves. If your model is wrong about facts, no amount of fine-tuning will reliably fix it. If it knows the facts but presents them badly, retrieval won't help.

When RAG is the answer

Retrieval-augmented generation fetches relevant documents at query time and puts them in front of the model. It's the right tool when your problem is knowledge.

  • Your content changes. Documentation, policies, and pricing update weekly — retrieval reflects that instantly; a fine-tune goes stale the day it finishes.
  • You need citations. RAG can point at the source document; a fine-tuned model cannot tell you where it learned something.
  • You need access control. Retrieval can filter by permission before the model ever sees a document. Weights cannot forget selectively.
  • You're early. RAG takes days to stand up and hours to change your mind about.

When fine-tuning earns its place

Fine-tuning adjusts the model's behaviour. It's the right tool when your problem is form rather than fact.

  • You need a rigid output format that prompting only mostly achieves — and "mostly" is breaking downstream systems.
  • You have a specialised tone or domain register that takes 800 words of prompt to describe.
  • You're paying for those 800 words on every single call, and volume makes that the dominant cost.
  • You have hundreds of high-quality examples of the behaviour you want. Not dozens.

If you can't produce 200 clean examples of the behaviour you want, you don't have a fine-tuning problem. You have a prompting problem.

The cost picture

RAG's costs are ongoing and per-query: embedding, retrieval, and a longer context on every call. Fine-tuning's costs are front-loaded — data preparation, training runs, and evaluation — with cheaper inference afterward because the prompt shrinks.

The crossover depends almost entirely on volume. At low traffic, RAG is cheaper in every dimension that matters. At high sustained volume with a stable task, a fine-tune's shorter prompts can pay back the training cost within months. Model the arithmetic for your actual traffic before assuming either.

Latency

RAG adds a retrieval hop — typically 50 to 200 milliseconds — plus the cost of processing a longer prompt. Fine-tuned models skip both. For most interactive applications this difference is irrelevant. For high-frequency automated calls, it isn't.

The honest recommendation

Start with RAG and a carefully written prompt. Measure. If accuracy on your evaluation set is the problem, improve retrieval — better chunking and reranking beat fine-tuning for knowledge problems nearly every time.

Only reach for fine-tuning when you can point at a specific behavioural failure that prompting cannot fix, and you have the examples to fix it. And note that the two aren't rivals: the strongest production systems we build often use a fine-tuned model for consistent behaviour and retrieval for current facts.