Context Windows: What Fits, What Gets Dropped, and Why It Matters
The context window is everything a model can use for one response. Anything outside it does not exist for the model at that moment.
On this page
A support session began with a refund policy, then moved through an hour of shipping questions, a billing dispute and a password reset. When the customer asked about the refund again, the answer came back slightly wrong. The model did not forget the policy in the everyday sense. The policy may have been outside the input for that call, or buried under so much other text that it carried little weight. The context window sets the boundary of what a model can use for one response, and it explains many failures that look like memory problems.
What the window holds
Everything the model reads while producing one response sits in the window. That includes the system instructions written by the developer, the definitions of any tools it may call, the conversation so far, any documents or search results pasted in, and the tokens the model has already written for this response. Output counts because each new token is generated with every earlier token as input. A fixed window is shared among all of these, so a long system prompt or a large set of tool definitions shrinks the room left for the user's material.
The limit is measured in tokens and it is a hard ceiling. Some systems refuse input that is too long. Others truncate it quietly, and truncation is the more dangerous case, because the response still looks complete.
Why text outside the window does not exist for the model
A model does not keep state between calls the way a person does. Each call starts from the input it receives. If a conversation feels continuous, the application is almost certainly re-sending earlier turns, or a summary of them, with each new message. When a product says it remembers something, the usual mechanism is that a stored note was added to the input for that call. The note is only as present as the space it occupies in the window.
This distinction matters when designing a feature. To make a fact available, put it in the window, store it where the application can fetch it, or both. Relying on the model to recall something from an earlier session means relying on an input that may not be there at all.
What long inputs cost
Length carries three costs. The first is money, since most hosted services charge for every input token. The second is time: a long input takes longer to process before the first output token appears, and output is then produced one token after another. The third is quality. Evaluations of long inputs have repeatedly reported that models use material near the start and end of a context more reliably than material buried in the middle. The size of that effect varies by model and task, so test it on your own material rather than assume it.
The computation inside the model also grows with length. In the standard design, the work of relating every token to every other token rises faster than the length itself. That is one reason providers bound their windows and why very long inputs are slow. The arithmetic is not needed to design around the problem. What matters is that doubling the input does not reliably double the usefulness of the answer.
Strategies and their trade-offs
When material does not fit, or fits but dilutes the question, several strategies are common:
- Summarize older material and keep the summary, accepting that details will drop out.
- Retrieve only the passages relevant to the current question. This is precise when retrieval is good and fails quietly when it is not.
- Split a large document into sections and process them separately. Cross-references are lost unless the important links are passed along.
- Keep a short structured record of decisions and open items, and pass that instead of the full transcript.
- Trim the history to recent turns. This is simple, but it discards early constraints the user still expects to be honored.
No single approach is right everywhere. A summary is cheap but lossy. Retrieval is precise but only as good as its search. Splitting runs in parallel but cannot see connections between parts. Choose the approach whose failure can be detected, and test it on the cases where early material matters most.
Checking what fits
A short procedure keeps surprises out of production:
- List every component that enters the window for a typical request, including the system prompt and tool definitions.
- Count each component with the provider's counting tool rather than estimating from word counts.
- Set an output budget and reserve that room before filling the input.
- Run the feature with the longest realistic input, not the average one.
- Test whether the answer draws on material at the beginning, middle and end of the input, by placing a known fact in each location.
- Log token counts for real requests so that a template change which quietly adds weight becomes visible.
The last step is the one most teams skip. A prompt grows through small additions over months, and no single addition seems large enough to matter until the window is nearly full.
A check for one feature
Pick a feature that uses long input and write out its window at the heaviest realistic load: how many tokens go to instructions, history, documents and expected output. Identify the single fact the feature depends on most. Then run it three times, placing that fact at the start, the middle and the end of the input, and note whether the answer changes and whether the model ever reports that it could not find the fact. The results show whether the feature needs a different arrangement, a retrieval step or a narrower scope. Keep the record, because the next template change will shift the balance again.
