Latency and Streaming: Why the First Token Matters

Two people can wait the same number of seconds for an answer and report very different experiences. One watched words appear as they were written. The other stared at an empty box. The gap between those experiences is shaped by one measurement, the time to the first token, and by the choice to stream output rather than hold it until the end. This article explains what drives each timing and how to design around both.

Two clocks, not one

Latency for a model call has at least two parts. Time to first token covers the interval from sending the request until the first piece of output arrives. Total time covers everything after that, until generation finishes. The two respond to different causes. Time to first token depends largely on reading the input. The model must process the whole prompt before it can produce its first output token, so long prompts and large amounts of retrieved context push it up. Total time adds the generation itself, which proceeds one token at a time, so long outputs take longer to finish.

A feature that returns a short label and a feature that writes a long report can share the same first-token time and differ greatly in total time. If you only measure one clock, you will misdiagnose the slowness.

Throughput, often quoted as tokens per second, describes the second clock as a rate. It depends on the hardware, the model, the load on the serving system and the length of the context, so a single figure is only a snapshot. Use it to compare setups under the same conditions rather than as a fixed property of a model.

What streaming changes

Streaming sends each token to the client as it is produced rather than holding the whole response until the end. The computation is the same and the total time is roughly the same. What changes is the moment the user first sees progress. For interactive features that moment often matters more than the finish line, because it tells the person that something is happening and that the answer is on its way.

Streaming has costs. The client has to handle partial text, render it safely and cope with a connection that drops halfway through. If the output must be parsed as data, the program cannot act on it until the stream completes, or it has to parse incrementally, which is harder to get right. For a background job that nobody watches, streaming adds complexity with little benefit.

Designing around the wait

Good interfaces treat waiting as part of the design. Show within a moment that work has started, even if the content is not ready. Display partial results when they are meaningful, such as the opening paragraph of a summary, and make clear that they may still change. Give people a way to stop generation, which saves time and money when an answer is already going the wrong way.

Decide in advance what happens when a stream fails midway. Showing a truncated answer as if it were finished is the worst option. A better one shows what arrived, marks it as incomplete and offers a retry. The same discipline applies to timeouts. A request that takes too long should end in a clear state the user can act on, not a spinner that never resolves.

Progress indicators need the same honesty. An indicator that advances on a timer, regardless of what is happening, can make a stall look like activity, and users notice when a bar says nearly finished and nothing arrives. Tie any indicator to something real, such as the tokens received so far or the stage of a multi-step job.

Measuring what people feel

Measure from the client, not only from the server, because network time and rendering both affect what the user experiences. Record both timings for each call, along with the input length, the output length and the model used. Keep the timing data alongside the results, so a slow answer can be matched with the request that produced it. Decide where the clock starts as well. It can start at a button press, at the moment the request is sent, or when the first byte arrives, and each gives a different number. The one users feel is the first, so report it alongside the others and say which start point you used.

Look at the spread rather than the average. A small number of slow responses dominate complaints, so track the high percentiles as well as the median. Measure under realistic load as well as in quiet testing, since queuing can add delay that a single test request never encounters. Put these measurements where other product metrics live, so that a change in the trend is visible after a release.

Practice: instrument one call

Choose a feature in a product you work on that calls a model. Add timing around the request, record when the first token arrives and when the last one does, and log the input and output sizes. Run the feature twenty times with a short input, then twenty times with a long one, and compare the two distributions of first-token time.

If the long input moves the first token much more than the short one, the cost lies in reading the input, and the usual remedy is to send less context. If the total time is the main difference, the cost lies in the output, and the remedy is to ask for less or to stream sooner. Write down which case you found. The diagnosis matters more than the fix, because a fix applied to the wrong clock wastes effort and leaves the complaint in place.