Step 6 of 10
Attention
A mouse that won't move: the animal or the gadget? How does the model know?
Make a guess
"My computer is on, but the mouse has stopped moving." How does the model work out which mouse this is?
- It looks up "mouse" in a dictionary and takes the first meaning.
- It looks at the other words in the sentence, like "computer".
- It doesn't: to the model, a mouse is always an animal.
Show the answer
The answer: It looks at the other words in the sentence, like "computer".
When it reads "mouse", the model turns to the words already written and draws the meaning from them. Here "computer" carries a lot of weight: this is the device.
What happens
Take the sentence "My computer is on, but the mouse has stopped moving." At first, "mouse" gets its usual position, the same for the animal and the device.
To settle it, the model has each word look back at the words before it. Each one shares out its attention among them and mixes what it finds into its own meaning.
Dozens of "heads" do this at the same time, each in its own way, and the model stacks dozens of layers of this calculation.
In Qwen3-4B, one of those heads gives Computed for this site by the open model Qwen3-4B, on November 7, 2026.Question asked: "My computer is on, but the mouse has stopped moving. What should I do?"Share of its attention that one head of the model gives to "computer" while reading "mouse".Layer 2, head 5, counting from 0: the clearest head in the model on this sentence. Trace ambigue.en. Model card of its attention to "computer" when it reads "mouse".
A word only looks back, at what's already written. Here "computer" comes before "mouse", so the model can use it.
Your turn
Change the layer and the head, and see which words "mouse" pays attention to. Compare a single head with the average of all of them.
You wrote your own question. What follows was computed ahead of time on a sentence from this site, by a model too heavy to run in a phone's browser.
My computer is on, but the mouse has stopped moving. What should I do?
We follow "mouse": it is the only word in the sentence whose attention was computed. In gray, the tokens that come after it: it cannot see them.
Layer 3 of 36, head 6
"mouse" gives 83% of its attention to "computer".
Elsewhere: the instructions and the conversation tags 12%
The heads that look most at "computer"
Computed by the open model Qwen3-4B, revision 1cfa9a7.
Takeaway
Each word looks at the words before it and uses them to pin down its meaning. That's attention: context changes what a word means.
Go further
Where the name comes from
Google researchers proposed building language models around attention, in a paper titled Attention Is All You Need. The architecture it describes, the transformer, is the T in GPT, and most of today's large models descend from it.
One sharp head among many
The head shown here is the sharpest in the model for this sentence. In most of Qwen3-4B's layers, no head looks at "computer" anywhere near this clearly. Nobody programmed what each head does: its role took shape during training, and researchers only partly understand it.
See a whole model
Brendan Bycroft's visualization lays out every calculation of a small model in three dimensions, attention included.
Sources
- Vaswani et al., Attention Is All You Need (arXiv 1706.03762), June 12, 2017
- Qwen, Qwen3-4B, examples computed for this site on November 7, 2026