What a Model Learns From Its Data, and What It Cannot Learn
Training data shapes what a model knows, how it writes and what it overlooks. The gaps are as informative as the strengths.
On this page
People often talk about a model as if it knew everything in its training material. That picture is close enough to be useful and wrong in ways that matter. A model is shaped by a very large body of text, and what it keeps is a compressed trace of that text, weighted by how often and how clearly things appeared in it. Understanding that trace explains why a model is fluent in some areas, unreliable in others, and silent about things that were never written down.
Coverage and skew
Training material is not a random sample of human writing. Some languages have far more published text than others. Some subjects, such as widely used software libraries or heavily documented history, appear in enormous quantities. Regional regulations, the procedures of small companies and specialized trade practices may appear thinly or not at all. Frequency also shapes what is learned. A fact repeated in thousands of places is represented more firmly than one mentioned once in an obscure document.
Languages add a further layer. A model may be fluent in a widely used language and noticeably weaker in a smaller one, because less text was available in the smaller one. Test in the languages your users actually write in, not only in the one your examples happen to use.
The result is skew. A model may describe a popular framework accurately and then produce plausible but invented details about an obscure one, with the same confident tone. The skew also reaches style. Writing patterns that dominate the data become the defaults a model reaches for, which is why its prose can sound similar across very different subjects.
Recency and cutoffs
Training data is collected up to a point and then frozen. The model has no knowledge of anything after that point unless the information is supplied in its input. Two consequences follow. Questions about recent events, newly released tools or changed rules will be answered from older material, often with no signal that the answer is out of date. And the edge is not sharp. Commentary about any period keeps accumulating for years afterward, so the final months before a cutoff are usually thinly represented. A model may know an older event in more detail than a more recent one, even though both fall before its cutoff.
Cutoffs also interact with the other gaps. A subject that changed after the cutoff, such as a tool whose interface was later redesigned, may be described through its old design, and the model may present that design as current. The safe assumption for any fast-moving subject is that the model holds a snapshot, and that the snapshot is older than the moment you are asking.
What was never written down
Some knowledge is never published as text. How a particular team makes decisions, the unwritten rules of a workplace, the reasons behind a legacy design choice and the contents of an internal document are all absent from public training data. Expert judgment is often the same. A seasoned practitioner may weigh a case in ways they could not fully put into words, and nothing in the training material records that weighting.
A model cannot learn these things from its training. It may guess at them, and a guess can sound exactly as fluent as a fact. The remedy is to supply the material in the input or to retrieve it at the moment of the question. Retrieval is covered in another article in this series. The point here is that the gap is structural, not a flaw a newer model will simply close.
Learning is compression, not storage
A model does not keep a file of facts that it reads back. Its behavior comes from adjusting a very large set of numerical weights so that its predictions fit the training material. Patterns that recur are reinforced many times. Details that appear rarely are represented weakly and can blur. This is why a model can give the general shape of a rare fact, such as the field it belongs to and its rough era, while getting the specific name or figure wrong.
For a reader, the practical lesson is that gaps are often patchy rather than total. A model may be solid on the core of a subject and unreliable at its edges. The edges are where a careful check should be concentrated.
Checking a claim about coverage
Claims that a model knows a domain should be tested rather than accepted. Build probes whose answers you can verify from sources the model could not have seen. Material published after its cutoff is one option, and internal documents you control and have never supplied are another. Compare performance across languages and across subjects that matter equally to you. Then look at the pattern of failure, not only the rate. Does the model decline, guess with confidence, or quietly substitute a more common answer? Substitution toward the familiar case is a sign of skew.
Be cautious with statements about what a model was trained on. Developers often do not disclose the full mix of sources, so the behavior you can observe is usually more reliable evidence than a description of the data.
Practice: design a coverage probe
Choose a domain you know well, ideally one where you can say exactly what a careful expert would get right. Write twenty questions in three groups: common knowledge, moderately specialized material, and detail tied to a local or recent context. Ask the model each question, record the answer and mark it against your own verified answer. Do not average the results across groups. Look at where the errors cluster and what kind of error each one is.
The pattern will tell you more than the score. It shows which tasks the model can be trusted to do alone and which need the relevant material supplied in the input. Write that split down as two short lists. It becomes the starting point for deciding where to add context and where to keep a person in the loop.
