Skip to content
AI Universe
Français

Step 7 of 10

The next word

Does the model know the answer before it writes it?

Make a guess

You ask an assistant like ChatGPT the exact same question twice. What happens?

  • It gives the same answer, word for word.
  • The answer often changes a little.
  • Same answer, unless something changed online in between.

Show the answer

The answer: The answer often changes a little.

No answer is stored anywhere. At every token, the model works out the odds of every possible next token, and the assistant draws one at random, favoring the likeliest. Two draws can head off in different directions.

What happens

  1. What comes out of the model is a list: every possible token, each with its odds of coming next.

  2. A real example, from the model Qwen3-4B. It has written "A plane can fly because its". Next, it gives Computed for this site by the open model Qwen3-4B, on November 7, 2026.Question asked: "How can a plane fly when it is so heavy?"Probability of the token "engines" right after "…because its", at temperature 1.Trace avion.en. Model card to "engines" and Computed for this site by the open model Qwen3-4B, on November 7, 2026.Question asked: "How can a plane fly when it is so heavy?"Probability of the token "wings" right after "…because its", at temperature 1.Trace avion.en. Model card to "wings". It picked "engines", and the explanation went wrong from there: engines push a plane forward, wings hold it up.

  3. It keeps one token, adds it to the end, and the whole model runs again to find the next one.

  4. No answer was sitting in a database, waiting for you. Yet in the US, US adults who think AI looks up the exact answer in a databaseTavern Research survey for Searchlight Institute, online panels, August 1 to 6, 2025, 2,301 US adults, margin of error 3 points; a question with four answers to pick fromSource: Searchlight Institute, Americans Have Mixed Views of AI and an Appetite for Regulation (December 16, 2025)Read at the source, from the organization that produced it. of adults surveyed think that's how it works.

  5. Assistants don't always take the favorite: they draw at random among the likeliest tokens. Ask the same question twice and the answer will often change.

Your turn

Pick one of the site's questions and have the answer written one token at a time, looking at the candidates at each step. Then slide the temperature: the lower it goes, the harder the favorite crushes the rest.

Why is the sky blue?

The answer, token by token

Nothing written yet.

Read the whole answer

The sky appears blue because molecules in the atmosphere scatter shorter wavelengths of light, like blue, more effectively than longer wavelengths.

The slider changes the probabilities, not the answer: the model wrote it by taking the most likely token every time, and that token stays on top at every temperature.

The 20 most likely tokens for token 1 of 25

  1. The> 99.9%, written
  2. .The< 0.1%
  3. The< 0.1%
  4. THE< 0.1%
  5. "The< 0.1%
  6. the< 0.1%
  7. This< 0.1%
  8. It< 0.1%
  9. Sun< 0.1%
  10. nThe< 0.1%
Show the next 10
  1. >The< 0.1%
  2. Sc< 0.1%
  3. (The< 0.1%
  4. :The< 0.1%
  5. Why< 0.1%
  6. That< 0.1%
  7. Because< 0.1%
  8. Light< 0.1%
  9. When< 0.1%
  10. The< 0.1%

All the other tokens in the vocabulary, together < 0.1%

Computed by the open model Qwen3-4B, revision 1cfa9a7.

Takeaway

The model doesn't know its answer in advance, and it doesn't look it up anywhere. It works out the odds of every possible token, picks one, and starts over: the answer is written one token at a time.

Go further

Likely isn't the same as true

The odds the model computes measure how much a continuation looks like the texts it read during training (see "Where its knowledge comes from"). They say nothing about truth. When a question looks like a lot of texts, the right token wins by a mile: after "The capital of Australia is", Qwen3-4B gives Computed for this site by the open model Qwen3-4B, on November 7, 2026.Question asked: "What is the capital of Australia?"Probability of the token "Canberra" right after "…Australia is", at temperature 1.Trace australie.en. Model card to "Canberra". When it looks like nothing the model has seen, the odds spread out, and it writes something anyway (see "When it gets things wrong").

Temperature

Before the draw, the assistant can sharpen or flatten the gaps between the odds. That setting is the temperature. Turn it down and the favorite wins almost every time, so answers repeat. Turn it up and long shots get through more often: more variety, and more mistakes. On this site, the answer shown always follows the likeliest token, the only path the model computed, so the slider changes the odds at each step but not the sentence. To watch another model write at whatever temperature you like, Transformer Explainer runs GPT-2 right in your browser. The model is a big download, so wifi helps.

Does it see past the next token?

Working token by token doesn't stop a model from aiming further ahead. Looking inside one of their models, Anthropic researchers found that before it writes a line of poetry that has to rhyme, it already has a few candidate words in mind to end it. That doesn't mean the sentence is written in advance, but computing one token can set up the next few.

All these numbers come from Qwen3-4B, the open model introduced in "Tokens".

Sources