Retrieval or Fine-Tuning: How a Model Should Learn Your Material
Many teams that think they need fine-tuning need retrieval first. The difference comes down to whether the knowledge changes and what behavior you need.
On this page
Someone suggests fine-tuning a model on the product documentation, and the idea sounds natural, since the documentation is the knowledge the assistant needs. The trouble is that fine-tuning changes how a model behaves more reliably than it changes what the model recalls. Retrieval changes what the model sees when it answers. Many teams that ask for fine-tuning to supply knowledge would be better served by retrieval, and the difference is worth understanding before any budget is spent.
Two different levers
Retrieval adds material to the input at the moment of answering. The application searches a collection, selects passages and places them in the window, and the model answers from them. The model's weights do not change. When the documents change, the collection changes, and the next answer reflects it.
Fine-tuning adjusts the weights using example inputs and desired outputs. It is effective at shaping behavior: tone, output format, the handling of a domain's conventions, the decision rules for a classification task, or a consistent style of answer. It is a weak and unreliable way to install a body of facts, for reasons that follow from how knowledge is stored.
Why facts in weights are fragile
Knowledge learned through training is spread across the weights and tangled with everything else the model learned. A fact may be recalled correctly in one phrasing and missed in another. The model usually cannot say where a recalled fact came from, so it cannot cite it. Updating a fact means producing new training data and training again, and removing a wrong fact is harder still. Once the documentation changes, a fine-tuned model begins to drift from it, and nothing inside the model signals the drift.
Retrieval avoids most of these problems. The source is explicit, an update is a change to a document, and a citation can point to the passage that was used.
The choice is not always exclusive. Some systems retrieve facts at query time and fine-tune for output format, for the vocabulary of answers in a domain, or for the decision rules of a classifier. The two levers answer different questions, and combining them is reasonable once each has been justified by a failure that only that lever can fix.
What retrieval needs to work
Retrieval has parts of its own, and most failures come from them rather than from the model. Documents must be split into chunks of a sensible size, with headings and titles kept alongside the text. A search method must find candidate chunks, often by combining keyword matching with vector similarity. A ranking step must put the right chunk near the top, and the application must pass the chosen text, with a source identifier, into the window. Each stage can fail.
The most useful diagnostic is simple. For a set of real questions, check whether the correct passage was in the retrieved set at all. If it was not, the model never had a chance to answer correctly, and fine-tuning would not have helped. If it was, the remaining errors are about use, and those can be examined directly.
Decision criteria
Several questions usually settle the choice:
- How often does the knowledge change? Frequent change favors retrieval.
- Does the answer need to cite its source? Retrieval makes citation natural.
- Is the failure about what the model says, or how it says it? Facts point to retrieval. Format and tone point to examples, fine-tuning or a better prompt.
- Is there a large, clean set of examples of the desired behavior? Without one, fine-tuning is guesswork.
- Who will maintain the training data and retrain on each change? The maintenance cost is often the deciding factor.
A test before committing
Run the cheapest experiment first, and escalate only when it fails. A workable sequence runs as follows:
- Assemble a few dozen real questions with reference answers, drawn from the work the assistant is meant to do.
- Establish a baseline with a clear prompt and no retrieval, and record the failures by category.
- Add retrieval over the documents, measure again, and check whether the right passages were found.
- For failures where the facts are present but the format or style is wrong, try a few worked examples in the prompt before any training.
- If behavior problems persist with the right facts in view, consider fine-tuning, and evaluate it against the same set.
Count the costs of each path as well as its results. Retrieval adds a search step and a larger input to every call, and it needs an index that someone keeps current. Fine-tuning needs curated examples, training runs, evaluation of each new version and a plan for retiring it. A path that wins on quality can still lose on total effort, and the comparison should show both.
The order matters because each step is cheaper than the next, and each one produces information that narrows the choice.
Practice: match each failure to a lever
List the failures from the baseline, one per line, and mark each as factual, format or judgment. Factual failures point toward retrieval, format failures toward examples or fine-tuning, and judgment failures toward clearer criteria or human review. Count each kind. If most are factual, the training budget is probably in the wrong place. Then write one sentence describing what would have to be true for fine-tuning to be the right choice for this case. If that sentence cannot be written, the answer for now is retrieval, or nothing at all.
