How a Language Model Chooses the Next Word

Give a model the start of a sentence, "The capital of France is", and it will almost certainly finish with "Paris". That looks like recall, and the answer is right, but the process behind it is narrower than it appears. Every sentence a model writes comes out of one repeated step: choosing the next piece of text. Seeing how that step works explains where a model will be fluent, where it will be wrong, and why one prompt can produce different words on different runs.

From characters to tokens

The model never works directly with letters or whole words. Text is first cut into tokens, pieces drawn from a fixed vocabulary that was settled before training began. A common word is usually one token. A rare name, a long identifier or a word from an unfamiliar language may be split into several pieces. Spaces and punctuation count as well, so "Paris" and " Paris" with a leading space can be different tokens.

This matters for everything that follows. The model's choices are made over tokens. When it stumbles on a string, the cause is often a split in an awkward place, not a failure to understand the string as a whole.

A score for every possible next token

After reading the tokens so far, the network produces one number for each token in its vocabulary. These numbers are called logits. A function called softmax converts them into probabilities that are positive and sum to one. Most of the probability usually sits on a small set of candidates. After "The cat sat on the", a word such as "mat" or "floor" might carry most of the weight, while an unrelated word like "philosophy" carries almost none, though never exactly zero.

The scores are not a lookup of facts. They are the product of weights learned from a very large body of text, shaped by the single goal of predicting what comes next. When the training text repeatedly placed a city after "the capital of", the network learned to score city names highly in that slot. It did not store a table of countries and capitals for later consultation.

Choosing one, then repeating

Something has to turn those probabilities into a choice. The simplest rule takes the most probable token every time, which is called greedy decoding. More often a system samples, drawing a token at random according to the probabilities, so a likely token is usually picked but not always. The chosen token is appended to the text, and the whole process runs again with the longer text as input.

Each choice is made locally, given everything before it. The model holds no separate plan for the paragraph as a whole. Still, its learned weights encode a great deal of long-range structure, which is why the output usually hangs together. Many hosted services let a caller adjust how much randomness the choice allows. Turning it down makes the choice more predictable, but it does not change the probabilities themselves.

Why fluent is not the same as true

The training goal rewards text that fits the patterns of the data. A date in the right slot, written in the right format and placed next to the right kind of noun, scores well even when the date is wrong. Ask for a citation and the model can produce something shaped exactly like one, with plausible authors, a plausible journal and a plausible year, because those are the patterns it has seen. Nothing in the process checks that string against the world.

The training objective is good at producing text a reader accepts. Accuracy is a separate property. It depends on whether the learned weights made the true continuation more probable than the false one, and the model gives no direct signal about which case it is in. Its confident tone comes from the same patterns that produce its correct answers, so tone alone does not reveal which case you are looking at.

Where the mechanism shows up

Several familiar behaviors follow from the loop:

• Repetition, because a phrase that has just appeared becomes more likely to appear again.
• Confident tone on uncertain facts, since the phrasing is high-probability whatever the facts turn out to be.
• Sensitivity to wording, because a small change in the prompt shifts the probabilities of every later token.
• Imitation of format, so a request that resembles a table, list or form tends to receive one.
• Drift over long outputs, where early choices constrain later ones in ways nobody planned.

Each is a property of the next-token loop rather than a separate defect, so the remedies are usually structural. Ask for a format a program can check, supply the facts the answer depends on, and verify the output instead of trusting its tone.

A small experiment to run

Choose a question with one correct answer and a plausible wrong one, ideally one you can answer yourself. Before asking, write down what you expect the model to say and how sure you are. Ask it three times with identical wording and note whether the answers match. Then rephrase the question twice, keeping its meaning, and see whether the answer moves. If it does, the mechanism is visible: the model is responding to the words it received, and those words are the only input it has.

Repeat the exercise on a question you cannot answer yourself, and check the claim against a source you trust. The aim is not to catch the model out. It is to learn which outputs can be accepted on sight, which need a check, and which need a source placed in front of the model before it answers.