Reasoning Models: More Thinking Time, Different Trade-offs
Some models spend extra steps working through a problem before they answer. That helps on some tasks and costs time and money on all of them.
On this page
Routing every ticket through a model that works through a problem before it answers produced an unexpected result. The labels arrived a little later, the bill rose, and accuracy on the simple categories did not change. The extra work was real, but the tickets did not need it. Some models produce additional text on intermediate steps before the final answer. That space can help a great deal or add cost and delay for no benefit, and the skill lies in recognizing which kind of task is in front of the system.
What the extra work is
A reasoning model generates intermediate text before its final answer. That text consumes tokens, takes time to produce and is billed like any other output. Within it, the model can break a problem into parts, try an approach, notice an error and return to an earlier step. Depending on the product, the intermediate work may be shown in full, summarized or hidden. Whatever its visibility, it is output the model produced, subject to the same limits as any other output. The model is not running a separate program to check its work, and it is not guaranteed to report faithfully what drove its answer. A written rationale is a useful aid for review, not proof of the process behind the answer. Check the answer against the same evidence used for any other output.
The model is trained to make productive use of that space, and the benefit is conditional on the task. Extra steps help when the steps are needed. They add noise and cost when they are not.
Where it helps
Extended working tends to pay off on problems with several dependent steps: multi-part arithmetic, logic puzzles with many constraints, planning where an early choice limits later ones, and debugging where the cause is several steps removed from the symptom. It helps when an answer must satisfy many conditions at once, because candidate answers can be checked against the conditions before one is committed. It also helps when a question contains a trap, such as a premise that needs testing before it is used. Code generation shows the trade clearly. A function with a precise specification and clear tests seldom needs long working, while a change that touches several interacting modules may benefit from it. The useful test is whether the extra steps changed the output for the better on the tasks where they were used.
Where it wastes effort
Simple tasks gain little. Looking up a stated fact in a short passage, classifying a message into a clear category, reformatting text, correcting spelling and writing a short reply are tasks where the answer sits close to the surface of the input. Extra steps there mostly add time. Clear rules help as well. If the decision is a lookup in a table supplied in the input, the model has little to reason about.
There is a subtler cost. Longer working can overthink an ambiguous case, reinterpreting a clear instruction until the output drifts away from what was wanted. Measure the change in accuracy rather than assuming that more thinking is better.
Latency, cost and variance
Longer working means longer waits and more billed output. Because the amount of intermediate text varies from request to request, the cost per call is less predictable than for a standard model. Some services allow a budget for the reasoning portion, and some do not, so check the controls available for each model. Set timeouts, and design the interface so that a slow answer does not look like a failure. If the output is streamed, the intermediate working may not stream in the same way as the final answer, so the interface should not assume it will. Measure the spread of response times, not only the average. A few long chains of working can dominate the experience for a small share of requests, and a timeout that suits the typical case may cut off the cases that most needed the extra effort. Decide in advance what happens to a timed-out request: retry at a lower setting, queue it, or report that no answer is available.
Choosing how much reasoning a task deserves
A practical method avoids both defaults, always-on and never:
- Assemble a labeled set covering the tasks the system performs, with several examples from each category.
- Run the set at the lowest reasoning setting the model offers, and record accuracy, latency and cost per item.
- Run the same set at higher settings, and record the same measures.
- For each category, find the lowest setting that meets the accuracy target, and note where higher settings improve results enough to justify their cost.
- Route each category to its setting, and recheck the routing whenever the model or the task mix changes.
The result is usually a mixed system, with high-effort reasoning on a few hard categories and fast responses everywhere else.
Practice: sort ten tasks before running them
Choose ten tasks from ordinary work and sort them, before running anything, by whether they need several dependent steps. Predict which ones will benefit from extended working. Then run all ten at two settings, recording accuracy, time and cost. Compare the predictions with the results. The cases where the prediction was wrong are the most valuable part of the exercise, because they show the kind of task that intuition misjudges. Write those cases down and keep them with the test set, since they will matter when the model changes.
