Temperature and Sampling: Controlling Variation, Not Truth

A common instinct is to set temperature to zero when accuracy matters. The result is often a model that gives the same answer every time, which feels reassuring until the answer turns out to be the same wrong one. Temperature changes how the model chooses among its candidate tokens. It does not change which candidates the model considers likely, and most errors come from that part.

What temperature does

Before a choice is made, the model's scores for the candidate tokens are divided by the temperature, and the results pass through softmax to become probabilities. A temperature below one sharpens the distribution, so the most likely candidates gain weight relative to the rest. A temperature above one flattens it, giving less likely candidates a larger share. As the temperature falls toward zero, the choice approaches the single most probable token at each step, which is greedy decoding.

Two things follow. Lower temperature makes outputs more repeatable across runs. It also adds no knowledge, because the underlying scores are unchanged. Temperature alters the selection rule, not the evidence.

Top-p and top-k

Two other settings restrict which candidates are eligible. Top-k keeps only the k most probable tokens at each step. Top-p, also called nucleus sampling, keeps the smallest set of tokens whose probabilities add up to at least p. When the model is confident, a top-p setting keeps very few candidates. When it is uncertain, the same setting keeps many. Both interact with temperature, so change one at a time and test the effect. Hosted services also differ in which controls they expose for which models, and some do not accept every setting, so check the documentation for the model in use.

The effect of these settings depends on the task. It is largest on open-ended work, where the tail of the distribution holds many acceptable continuations, and smallest on tasks with one correct answer, where the top candidate already dominates the rest.

Repeatable is not correct

Suppose a model assigns the highest probability to a wrong date. At zero temperature it will return that date every time. The stable output may look like evidence, but the evidence concerns the process, not the content. A stable wrong answer is still wrong.

The repeatability from lower temperature is useful when the same input must yield the same output, for example in a test suite or a pipeline where downstream code expects consistent formatting. It is not a guarantee of factual reliability.

A related technique runs the same question several times at a moderate setting and keeps the answer that appears most often. On tasks with one correct answer, scattered errors tend to disagree with one another while correct answers agree, so the majority can be more reliable than any single run. The method multiplies cost and latency, and it cannot rescue a question on which the model's most probable answer is wrong across most runs. Temperature cannot fix that case either, for the same reason.

When variation helps and when it hurts

Variation is an asset when the task has many acceptable outputs. Brainstorming names, generating varied example sentences for a quiz, or drafting alternative phrasings of a message all benefit from some randomness, because a single deterministic answer gives one option where several were wanted.

Variation hurts when there is a right output and it is needed every time. Extracting a field from an invoice, classifying a support ticket into a fixed set of labels, or producing JSON that a parser will read are tasks where a different answer on each run means a different result in production. Low randomness is a sensible default for these, but it should be paired with validation and with a measurement of accuracy, not just consistency.

Several misconceptions are common:

• Zero temperature guarantees a correct answer. It guarantees only a repeatable choice.
• Higher temperature makes the model more creative in a way that improves quality. It widens the spread of outputs, which may or may not be better.
• A setting chosen once stays right. The right value depends on the task and should be rechecked after model updates.
• Identical settings always give identical output. Hardware differences, batching and provider updates can still change results, and some services describe seeds as best effort.

Testing repeatability and accuracy separately

Measure the two properties apart. To test consistency, run the same input many times at the intended setting and count how often the outputs agree, using the comparison that matters to downstream code: exact match, or agreement on the fields that count. To test accuracy, score the outputs against labeled answers. A setting can be highly consistent and inaccurate, or variable and accurate on average. Both numbers are needed to choose.

Record the model version, the settings and the date of each test. A change in consistency after a provider update is information, and it cannot be interpreted without that record.

A small comparison to run

Assemble a labeled set of about twenty items, such as short messages to be sorted into three categories. Run the set at two temperature settings, several times each. Record agreement across runs and accuracy against the labels. Then examine the items where the two settings disagree. Are the lower-temperature answers consistently wrong for one category, or wrong only on items near a boundary? The answer shows whether the setting is the problem or whether the labels need a clearer definition. In most cases the labels are the cause, and the setting turns out to have been a minor lever.