Why Models Make Things Up, and What Reduces It

Ask a model for the title of an obscure but real book and it may return a title that sounds exactly right and does not exist. The model has not decided to deceive anyone. It has produced the most plausible continuation of the question, and for this kind of question the plausible continuation and the true one can diverge. Understanding why that happens is the first step toward reducing it.

Plausible is the target

Training optimizes a model to predict text well, so text that fits the patterns of the data is what it learns to produce. It has no internal index of verified facts that it consults before speaking. Some facts appeared many times in the training text and are held in the weights in a robust way. Others appeared rarely, or appeared alongside conflicting details, and the weights reflect that weakness. When a question concerns something the model has seen little of, the patterns that remain are generic, and generic patterns are easy to fill with plausible detail.

Asking the model how sure it is does not solve this reliably, because the stated confidence is produced by the same process as the answer itself.

Training and evaluation also tend to reward complete, specific answers over answers that stop short. A gap in knowledge is therefore often filled rather than reported. The filling is fluent, which is what makes it hard to notice.

Where errors cluster

Errors are not spread evenly. They gather in a few predictable places:

• Rare names of people, places, products and organizations.
• Exact numbers, dates, version strings, quotations and statistics.
• Citations, page numbers and identifiers, which have a regular shape the model can imitate.
• Events after the end of the training data, which the model may describe from older patterns.
• Questions built on a false premise, which the model may accept and elaborate.

These are the details people are least able to verify by glancing at an answer, so they do the most damage. A paragraph that is mostly right with one invented figure is harder to catch than an answer that is plainly wrong.

Grounding the answer in sources

The most effective measure is to change the task. Instead of asking the model to answer from memory, supply the source material in the input and ask it to answer only from that material. A retrieval step can find the relevant passages in a larger collection, and the application passes them in. The instruction should say what to do when the sources do not contain the answer, because without it the model will often fill the gap from memory anyway.

Consider a policy document that states a refund window for one product line, and a question about a different line that the document does not cover. Grounded and instructed to decline, the model should say the document does not address that product. Ungrounded, it will often describe a typical refund policy, which reads as authoritative and appears nowhere in the source. The gap between those two outputs is the reason the measure is worth taking.

Grounding is not a cure. If retrieval misses the right passage, the model is answering from the wrong evidence, and it may still sound confident. Grounding moves the risk from memory to search. Search is easier to inspect, which is a real gain, but the risk remains.

Citations as a check, not a guarantee

Asking for citations can help, provided the limits are understood. A model may cite a source that does not contain the claim, or invent a source entirely. A citation is a pointer that must be checked, not evidence in its own right.

Check mechanically wherever possible. If an answer quotes a passage, confirm that the quoted words appear in the source. If it points to a section, confirm that the section exists and says what the answer claims. Automated string matching catches fabricated quotations. A reviewer, or a second model pass, can catch claims the quoted text does not support. A second model is also fallible, so check its verdicts against a sample of human judgments before relying on them.

Permission to decline, and verification

Designing for honest failure is part of the work. Give the model an explicit way to say that the sources do not answer the question, and treat that answer as a normal outcome in the interface. Then measure both kinds of error: answers that are invented, and answers that decline when the sources do contain the answer. A system that declines too often fails in a different way, and users learn to ignore it.

Numbers deserve special handling. If the model computes a total, recompute it in code. If it extracts a date, check that the date is valid and falls in the expected range. Deterministic checks are cheap, and they catch a class of errors that no amount of prompting removes.

Practice: measure the error rate on your own material

Build a set of twenty questions whose answers appear in a document you control. Run the set twice: once with the model answering from memory, and once with the document supplied along with an instruction to decline when the answer is missing. Count the fabricated details in each run and sort them by type: wrong numbers, invented names, false citations, and confident claims the source does not support. Then examine the grounded failures. Did retrieval miss the passage, or did the model misread a passage it received? The breakdown shows which measure deserves investment next, which is more useful than general advice about hallucination.