Reading a Model Card Like an Engineer

A model card is the document that a model's makers publish to describe what the model is for, how it was evaluated and where it falls short. It is a useful starting point and a limited one. Read it the way you would read a specification from a supplier. Note what is claimed, note the conditions attached to each claim, and note what has been left out. The gaps are often more useful than the highlights.

What a card usually contains

Typical sections include a description of the model and its intended uses, a summary of the training data at a high level, evaluation results, known limitations, safety evaluations, and terms of use or a license. The depth varies a great deal. Some cards list benchmark scores with no conditions attached. Others describe test procedures at length. Not every card has every section, and a missing section tells you something too.

Pay close attention to the sections on intended and out-of-scope uses. Out-of-scope uses are often listed in a sentence or two, and they show where the makers did not want the model relied on. A card that names no out-of-scope uses has told you less about its boundaries, not more.

Questions to ask of each claim

Start with intended use, and ask whether your use falls inside it. A model described as suitable for summarizing general text may be a poor fit for dense regulatory filings, even if it scores well on general tests. Next ask under what conditions each evaluation was run: which prompts, which settings, which version of the model and which grader. A score without those details is a number you cannot reproduce.

Then read the limitations section closely. Limitations are the best map of where to test, and a vague limitations section is a reason to test more, not less. Ask what the card says about data: what kinds of sources, which languages, what time period, and how personal data was handled, at whatever level of detail it offers. Finally, read the terms. Check whether commercial use is covered, whether your inputs may be retained, and whether the terms can change without notice to you. If your users write in several languages, check whether the evaluation covered each of them. A result in one language does not reliably transfer to another, and a card that reports results for one language alone has answered a narrower question than it may appear to.

What is usually missing

Some things rarely appear in a card. Performance in your language or your domain is often absent, because the card is written for a broad audience. Latency and cost under your own load are generally not given. Examples of typical failures are sometimes omitted, though they are among the most useful evidence a reader can have. The card also describes the model as of a particular date, and models change after release or are replaced by newer ones. Comparisons with other models need the same care. A card that compares itself with an older or smaller model shows the direction of change on the makers' chosen tests. It does not show the size of the gap for your work, and those tests may have been chosen because the model did well on them.

Safety work is often described in general terms without stating what was tested or how the results were judged. None of this makes a card dishonest. It means a card is a summary written for many readers, and your decisions need narrower evidence than a summary can provide.

Turning claims into tests

Treat each substantive claim as a hypothesis. A statement such as "handles long documents well" becomes a test: a set of long documents from your own collection, a question with a known answer for each, and a pass criterion written before you run anything. A stated limitation becomes a probe: does the failure appear in your inputs, and how often? Keep a short record for each claim:

• the claim, quoted as it appears in the card
• the conditions the card says it depends on
• the test you wrote and the pass criterion
• the result, with the model version and the date

These records matter later. When the card is revised or the model is replaced, you can see which claims you have already checked and which still need work. Dates matter for this record. When a card is revised, older claims may no longer describe the current version, so a result that held last quarter may not hold now. A test suite you can rerun is the cheapest way to find out.

Practice: write a test plan from a card

Choose a model card or specification sheet for a model you use or are considering. On a single page, list five claims that matter for your work, the condition each one depends on, and a test for each. Then list three things the card does not say that you need to know, and write a test for each of those gaps.

Compare the finished plan with what you would have checked if you had read only the summary at the top. The difference shows how much the card helped and where it stopped helping. If the plan is mostly tests for things the card never mentions, that is not a failure of your reading. It is the normal shape of a careful evaluation.