Evaluating a Model on Your Own Task Before You Trust It

A model's public score is an answer to a question someone else asked. One team chose a model from a leaderboard, shipped a feature built on it, and found that it struggled with their invoice layouts, which resembled nothing in the test set. The score was accurate. It simply described a different task. Public benchmarks answer a general question, and a product asks a specific one. Only a small evaluation built from the product's own inputs can answer that.

Public tests measure public tasks

A benchmark is a fixed set of questions with a scoring rule, and a model's score is its performance on that set. The questions were chosen for reasons that may not match the product. Their wording may differ from what real users write, their answers may be formatted differently, and their difficulty distribution may bear no resemblance to actual traffic. A high score shows that a model handles that kind of problem. It does not show that the model handles this one.

Public results still have a use. They can narrow a long list of candidates to a few worth testing. The mistake is treating the ranking as the decision.

Start from real inputs

The raw material for an evaluation is what the system will actually receive. If there are logs from a prototype or from a manual process, sample from them. If not, ask the people who handle the work to write examples. Include the common cases, the awkward ones and the known failure modes. Messy inputs matter: typos, missing fields, mixed languages, long attachments and scanned pages. Remove or pseudonymize personal data before the set goes beyond the people who need it.

Aim for coverage rather than size at the start. Thirty cases that span the real range will teach more than three hundred near-duplicates.

Define good before you grade

Every item needs a definition of a good output before anyone grades it. For some tasks this is simple: an extracted total is either correct or it is not. For others, such as summaries or drafted replies, write a rubric with explicit criteria, such as "states the decision, cites the relevant clause, and uses nothing outside the document." A person can check a criterion like that in a minute. Write one or two examples of excellent and unacceptable outputs for each criterion, so that graders share a standard.

Decide in advance what counts as a failure of each severity. A wrong amount on an invoice may be a hard failure, while an awkward phrase is a minor one. Weighting errors by their impact tells a team far more than a single pass rate.

Choosing a grader

There are four common ways to grade:

• Exact or programmatic checks, such as matching a value, validating a schema or recomputing a sum. These are fast and reliable where they apply.
• Human review against the rubric. This is the most trustworthy method for nuanced tasks and the slowest.
• A model acting as a judge, working from a rubric. It scales well, but it has to be calibrated. Compare its grades with human grades on a sample, and check whether its errors are systematic.
• Reference comparison, where a trusted answer is kept and outputs are compared with it by a defined method.

Most useful evaluations combine these. Programmatic checks carry as much of the load as they can, and human review is reserved for cases that need judgment or where automated checks disagree.

Tracking regressions

An evaluation earns its keep when it runs on every change. Record the model version, the prompt, the settings and the date with each run. Keep the set stable, adding cases as failures are discovered but never quietly editing old ones, so that results stay comparable. Look at results by category as well as in aggregate. A change that improves the average may break the one category that matters most to customers.

Sampled outputs vary, so a single run can differ from the next on the same input. Repeat runs on the cases that matter, and treat a change that is small relative to that variation as unproven.

Keep the evaluation set out of every prompt example and every tuning run. Once a case has been used to fix a failure, it no longer measures ability on unseen work. Keep it in the set, but tag it as a regression case, so that the headline figure is not inflated by cases the system was shaped around.

Practice: write the first twenty cases

Work through the first evaluation in this order:

1. Write the task in one sentence, together with the output the system must produce.
2. Collect about thirty real inputs with help from someone who knows the work.
3. Write the rubric and the severity levels before running anything.
4. Have two people grade a subset separately, and record each disagreement in writing.
5. Run the candidate systems on the full set, and read the failures before looking at totals.
6. Add the failures to the set, and rerun whenever anything changes.

Step four is the one teams skip, and it is the most valuable. Two graders who disagree have found a gap in the definition, and closing that gap is worth more than any leaderboard position. Before running any model, write twenty cases for a task you know well, and include at least four that you expect a model to fail. Ask a colleague to grade three of them without seeing your answers. Where their judgment differs from yours, rewrite the rubric until the difference is resolved or explained. The exercise takes an afternoon and produces the artifact that every later decision about models will depend on.