Fine Tuning vs RAG: Choosing Custom AI Architecture
Fine-tuning changes how a model behaves. Retrieval changes what it can see. Most teams that pick the wrong one are fixing a behavior problem with retrieval or a knowledge problem with training. This guide ties each choice to the failure you observe, with the research behind it, and covers two cases where neither is the right tool.
Start from the failure, not the technique
Run your off-the-shelf model on fifty real questions and sort the bad answers. Three piles come up again and again:
- Wrong or missing facts. The answer needs information the model does not have or that changed since training.
- Wrong form. The facts are fine, but the output breaks your JSON schema, ignores your tone, or misuses your terminology.
- Wrong scope. The question needs a synthesis across a whole corpus, or a decision that depends on who is asking.
Facts point to retrieval. Form points to fine-tuning, or often to a cheaper fix described below. Scope points to something else.
Why fine-tuning is a poor way to add facts
Training on your wiki nudges the model toward text that resembles it. It does not give the model a lookup table. The research is blunt about the result. In Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (Gekhman et al., EMNLP 2024), examples that introduce new knowledge are learned much more slowly than examples consistent with what the model already knows, and once they are learned they increase the model's tendency to hallucinate. A model fine-tuned on facts it did not previously know learns to answer confidently about things it cannot ground.
Direct comparisons point the same way. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs found that unsupervised fine-tuning gave only modest gains while retrieval consistently outperformed it, for knowledge the model had seen in training and for entirely new knowledge.
Compliance adds a practical reason. You cannot delete a deprecated document from a model's weights. With retrieval, you remove it from the index and it is gone from the next answer.
Long context is not a substitute for retrieval
When the corpus is small, putting everything in the prompt is tempting, and for a short manual it can work. The pattern degrades as the context grows. Lost in the Middle found that performance is often highest when relevant information sits at the beginning or end of the input and degrades significantly when it sits in the middle. You also pay for every token on every call. Retrieval puts the relevant passage near the question and keeps the prompt short.
What retrieval does not fix
Retrieval is not a universal answer to knowledge problems. Two cases deserve attention before you commit.
Global questions. "What are the recurring themes across all our customer contracts?" asks for a synthesis of an entire corpus. Top-k retrieval returns a handful of chunks, so the answer covers whatever the nearest neighbors happened to contain. The GraphRAG paper, From Local to Global, states the problem directly: RAG fails on global questions directed at an entire text corpus, and such questions are better treated as query-focused summarization than as lookup. For these workloads, plan for precomputed summaries or a map-reduce pass over the documents. Do not tune the chunk size.
Retrieval that cannot find the answer. If the right passage is not retrieved, no model can use it. Measure retrieval alone (did the correct chunk appear in the top results) before you judge the generator. Our guide to RAG pipeline evaluation covers how, and permission-aware retrieval covers filtering results by who is asking, which fine-tuning cannot do at all.
What fine-tuning is good for
Fine-tuning earns its cost when the problem is form, not knowledge: a consistent voice, strict structured output, a classification or routing step, or a specialized notation. It can also let a small model do a narrow job a large one does today, lowering per-call cost at volume. Before you start, check whether a cheaper tool solves it. Structured-output modes and schema validation with a retry fix many JSON problems. A few well-chosen examples in the prompt often fix tone.
If you do fine-tune, budget for the full cycle: curating thousands of clean examples, tuning hyperparameters to avoid forgetting the base model's abilities, running a held-out evaluation, and repeating when the base model is retired. The training data is part of your system and needs the same ownership as code.
A third lever: check the output deterministically
Some requirements are neither knowledge nor style. They are rules the output must obey, such as using the approved translation of a product term. Retrieval plus a deterministic check handles these better than training, because the check can fail a bad answer.
Retrieval does not remove the need for the check. The 2025 paper Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation injected noise into the retrieved context of a retrieval-augmented translation system. It found that low-resource language pairs, which lean harder on the retrieved text, degraded more severely and often produced nonsensical output. Reasoning models showed no improvement in error correction and were more susceptible, tending to rationalize the wrong context, and their confidence rose as accuracy fell. The authors conclude the field needs self-verifying integration. In practice that means a deterministic validator between retrieval and use. For terminology, check that the approved term appears in the output (with a deliberate allowance for inflected forms), and reject and regenerate when it does not. The retrieved text is not right because it was retrieved. The check proves it.
Decision table
| What you observe | First lever | Why |
|---|---|---|
| Answers wrong or stale on company facts | Retrieval | Updates by editing the index, citations possible, deletable |
| Right facts, broken JSON or wrong tone | Structured output, examples, then fine-tuning | Behavior is what training changes well |
| Must use exact approved terms | Retrieval plus a deterministic check | Verifiable after generation |
| Question spans the whole corpus | Precomputed summaries or map-reduce | Top-k retrieval cannot see the whole |
| Different users may see different documents | Retrieval with permission filters | Weights cannot enforce access |
| Narrow high-volume task on a costly model | Fine-tune a smaller model | Per-call cost |
Combine them when each problem is real
A hybrid fits when you have both a knowledge problem and a form problem. A fine-tuned model produces the structure and routing your downstream systems need, and retrieval supplies the facts. Build the retrieval baseline first, because it is faster, cheaper, and tells you which failures remain. Only then collect training data for what is left.
Verify the choice
- Baseline. Build the simplest retrieval pipeline with a standard embedding model and naive chunking. Score it on your fifty questions.
- Diagnose. If the wrong chunks come back, improve retrieval with metadata filters or hybrid search. If the right chunks come back and the answer is still malformed, you have a form problem.
- Cost. Measure tokens per answer. If context cost dominates at your traffic, test a smaller fine-tuned model for that step against the baseline.
- Hold out. Keep a test set that neither retrieval tuning nor training data has seen.
Next actions
Write down the three most common failures from your fifty-question run and map each to a row of the table. Build the retrieval baseline for the facts row. Collect fine-tuning data only for failures that survive it.
References
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? shows new-knowledge examples are learned slowly and then raise hallucination rates.
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs compares unsupervised fine-tuning with retrieval for adding knowledge.
- Lost in the Middle: How Language Models Use Long Contexts shows weaker use of information placed in the middle of long inputs.
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization describes RAG failing on global questions about a whole corpus.
- Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation shows noisy retrieved context degrading output, worst for low-resource pairs and reasoning models.