Why the Same Prompt Can Return Different Answers

Send the same prompt twice and you may get two answers that differ in wording, emphasis or even substance. That is often surprising, because most software returns the same output for the same input. A model is different. Its output is produced by sampling, and by a long chain of numerical computation that can vary with conditions. Knowing the sources of variation lets you decide which ones to control, which to tolerate, and how to test in spite of them.

Sampling is designed in

At each step, a model produces a probability distribution over possible next tokens. Sampling picks one of them. The temperature setting reshapes that distribution. Lower values concentrate probability on the likeliest tokens and make choices more predictable. Higher values spread probability more evenly and allow less likely choices. Top-p limits sampling to the smallest set of tokens whose probabilities add up to a threshold.

The consequence is path dependence. Once one token differs, everything after it is conditioned on that difference. A small change at the third word can produce a noticeably different paragraph by the thirtieth. This is not a bug in the sampler. It is what sampling means when the output is a sequence.

Some products also generate several candidate answers and pick one, or let the user regenerate until a satisfying one appears. Each extra candidate is another source of variation, and the user may not know that the answer they kept was one of several. If your product works this way, say so in the interface or the documentation.

Settings are not always the whole story

Setting temperature to its lowest value is often assumed to remove variation. It reduces variation, but it does not always remove it. Arithmetic on parallel hardware can produce tiny numerical differences, and those can flip a choice when two candidates are nearly tied. The way a provider groups requests together for processing can also influence the arithmetic.

Some interfaces offer a seed parameter intended to make sampling reproducible. Support, guarantees and conditions vary, so read the documentation for the exact service you use. Treat "deterministic" as a claim to test, not a property to assume. Output filters applied around a model can also change between runs or versions, and they may alter a response without any change to the model itself. Log the final text the user saw, not only the raw model output, so you can tell which layer produced a given result.

Changes outside your control

The model behind a name can change. Providers update models, sometimes under an alias that points to the newest version in a family, and behavior can shift even when your own code has not changed. Serving hardware and software change too. Pinning to a specific version identifier, where the provider offers one, reduces this, but it does not eliminate every change, and pinned versions eventually retire.

Log the model identifier and the date with every result. When an output changes without explanation, you can then check whether the change lines up with a known version change or with something in your own system.

Context differences

Often the variation is in the input, not the model. The system prompt may have been edited. A retrieved document may have ranked differently. The conversation history may be longer than it was. The prompt may contain a timestamp or a random identifier that changes on every run. Build the full input for each call, log it, and compare inputs before blaming the model. Many apparently random differences turn out to be differences in what the model was shown. Tool results are a common hidden input. If the assistant searches a database or calls an outside service, its answer can change because the underlying data changed between runs, not because the model did. Record or freeze those results when you need to reproduce a run.

Testing and explaining variation

Regression tests for generated text should check properties rather than exact strings. Useful properties include whether the answer contains the required fields, whether a cited source exists, whether a total matches its line items, and whether the tone stays within an agreed range. Run each test case several times and set an acceptable pass rate, rather than demanding that every single run be identical.

When you explain variability to users, present it as a property of the feature. Offer a way to regenerate. Keep the chosen version visible, so people can compare. Say plainly when an answer is an estimate.

Keep a reference output for each important prompt, saved on the day you first accepted it. A reference gives later comparisons something firmer than memory, which tends to hold on to the best answer rather than the typical one.

Practice: sort the variation

Take one prompt you rely on and run it twenty times with the same settings. Sort the outputs into groups according to what changed: wording only, a fact, the structure, or a refusal. Each group tells you something different. Wording changes are usually harmless. A changed fact is a quality problem. A changed structure may break code downstream, and a changed refusal may signal a shift in the boundary.

Then write the property test that would have caught the differences that matter and would have ignored the rest. That test is the real output of the exercise. It turns an unpredictable system into one whose acceptable variation is written down.