Getting Structured Output From a Model Reliably

A chat reply is written for a reader. A program needs something else: a shape it can predict, with fields in the right places and values of the right type. Getting a model to produce that reliably takes more than asking for JSON politely. It takes constraints, checks and a plan for the times the check fails. This article walks through each layer and where each one stops helping.

Three ways to ask for a shape

Most approaches fall into three groups. The first is instruction: describe the format in the prompt and show an example. This is flexible and cheap, but nothing enforces it. The model can drop a field, add a sentence of commentary before the object, or invent a key that nobody asked for.

The second is constrained generation, where the output is limited, at the moment each token is chosen, to text that keeps the result valid against a schema. Where it is available it removes many syntax errors. It does not guarantee sense. A date can be perfectly well formed and still be the wrong date, and a number can fit its type while being far from the true value. The third is tool or function calling, where the model fills in arguments for a function you declare. The shape is described as a schema, and the call is treated as a request to be checked before anything runs. Availability and details differ between providers, so read the documentation for the service you use rather than assuming one approach is universal.

Validate what comes back

Treat every structured output as untrusted input until it passes checks. Parse it first. Then validate it against the schema: required fields present, types correct, enumerated values within the allowed list. Then check the rules a schema cannot express. An end date that falls before a start date, a total that does not equal the sum of its line items, or an identifier that does not exist in your system are all valid JSON and all wrong.

Keep the validator separate from the prompt so you can test it alone, with deliberately bad examples. Record why each response failed. Those records are data about the model's weak spots, and they tell you which part of the prompt or schema to tighten next.

Normalize before you validate. Trim stray whitespace, pick one casing convention for enumerated values, convert dates to a single format, and decide whether unknown fields are an error or are dropped. Doing this in one documented step keeps the validator honest, because it is clear what counts as a valid value and what was quietly repaired.

Retries and their limits

When validation fails, a common response is to ask the model again and include the validation error, so it can correct the specific problem. This works often enough to be worth doing. Keep the retry message short and specific. Restating the whole task on every attempt often invites a different answer rather than a correction, and it spends tokens on instructions the model has already followed. Limit the attempts, usually to a small fixed number, and log each failure. Retries cost time and money, and they can loop on inputs the model cannot handle at all. A retry should not change the meaning of the task. If the same input fails several times in a row, stop and route it somewhere a person or a different process can handle it.

Designing the fallback

Decide before you ship what happens when an output cannot be made valid. The options include a default value that is clearly marked as defaulted, a queue for a person to review, a request to the user for the missing information, or a failure state that the calling code must handle explicitly. The wrong choice is to pass a partial object downstream and let later code guess at what was meant.

Make fallbacks visible in the data, so you can see how often they happen. A rising fallback rate often signals a change in the inputs or in the model version, and it is one of the cheapest early warnings a structured pipeline can have.

Keep the schema itself in version control alongside the code that reads it. The schema is part of the interface between the model and your program, and a change to one side without the other is a common cause of silent breakage.

Testing across messy inputs

Test with inputs that do not look like the examples in your prompt. Include empty documents, very long ones, text in an unexpected language, fields that contradict each other, numbers written as words, and content that contains instructions addressed to the model. That last category matters. Text inside a document can steer what gets extracted, so the test should check that the output still follows the schema when the input tries to pull it off course. Include a few inputs that are simply hard, such as a document whose key figure appears twice with different values. The right output may be a flag for review rather than either value. A test that expects a single correct answer will fail on those cases, and the failure will be informative.

For practice, pick an extraction task you care about and write its schema. Create fifteen test inputs across the categories above, including five with deliberately broken or misleading content. Run them through the pipeline and record the validation pass rate and the kinds of failure. Then decide which failure you would rather have, a visible fallback or a wrong value that looks right, and make the validator enforce that choice. The decision is the part that transfers to other systems.