Skip to content
AI Universe
Français

Step 2 of 10

Tokens

The model doesn't read your words. What does it see instead?

Make a guess

The model cuts the question "Why do cats purr?" into pieces. How many does it see?

  • One piece per word.
  • Fewer pieces than words: it groups them.
  • More pieces than words.

Show the answer

The answer: More pieces than words.

Common words like "do" and "cats" fit in one piece each. "purr" is rarer, so it gets cut in two, and the question mark is a piece of its own.

What happens

  1. Before any calculation, your question is cut into pieces called tokens. Here is "Why do cats purr?" as the model Qwen3-4B cuts it: Whynumber 10234 donumber 653 catsnumber 19423 purnumber 3999rnumber 81?number 30

  2. Under each piece is its number. That's its place in the model's vocabulary, a list of pieces fixed once and for all, before training even began.

  3. The model will never see your letters. All it gets is this string of numbers.

  4. Common words fit in a single token. Rare words, names and long words often get split into several.

  5. Language changes the count. The same question takes Counted by the tokenizer of the open model Qwen3-4B, on the question alone.Question asked: "Why do cats purr?"Trace chat.en. Model card in English and Counted by the tokenizer of the open model Qwen3-4B, on the question alone.Question asked: « Pourquoi les chats ronronnent-ils ? »Trace chat.fr. Model card in French: "Pourquoi les chats ronronnent-ils ?".

Your turn

Pick one of the questions and switch it between English and French: watch the number of tokens change. Then try a sentence of your own, in any language you like.

Tokens in

Why is the sky blue?

6 tokens

  1. Whynumber 10234
  2. isnumber 374
  3. thenumber 279
  4. skynumber 12884
  5. bluenumber 6303
  6. ?number 30

6 tokens in English, 10 in French: French takes 67% more.

Across the fourteen prepared questions (the twelve from the start and two that come up later): 114 tokens in English, 155 in French, 36% more for French.

Your own sentence

This is Qwen3-4B's tokenizer: about 2 MB to download, 11 MB once unpacked. Nothing downloads until you click. It comes from this site, and your sentence never leaves your device.

Splitting your own sentence needs JavaScript: the tokenizer runs in your browser.

Qwen3-4B tokenizer, revision 1cfa9a7, Apache 2.0 license. Read the license

Takeaway

The model reads neither letters nor words, but tokens: pieces of text, each one known by a number. The same question often takes more of them in French than in English.

Go further

Why some languages cost more

A tokenizer is built from a large sample of text: strings of letters that come up often in it become whole tokens. A language that's less common in the sample gets cut into smaller pieces. More tokens means more computing for the same idea, a slower answer, and less text the model can take in at once. For some pairs of languages, one study measured up to Gap in token count between two translations of the same text, in the most extreme cases measuredTokenizers of common language models in 2023, on the same text translated into different languages; tokenizers built for several languages still show gapsSource: Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (arXiv 2305.15425) (October 20, 2023)Read at the source, from the organization that produced it. as many tokens for the same text in translation.

One number per piece

A piece keeps its number from one question to the next. Look at "Why" in "Why is the sky blue?": Whynumber 10234 isnumber 374 thenumber 279 skynumber 12884 bluenumber 6303?number 30

Most pieces start with a space, and the space is part of the token. So a word at the very start of a text and the same word in the middle of a sentence don't always get the same number.

The model behind these examples

The splits, the odds and the attention shown on this site were computed by Qwen3-4B, an open model released by Alibaba Cloud. Open means anyone can download it and look at its calculations, which the models behind the big assistants don't allow. Yet ChatGPT, Claude, Gemini or Copilot work the same way, with much bigger models and extra steps, like searching the web.

Your own question is used at the start, here, and on the way back. For the steps in between, Qwen3-4B is too heavy to run in a phone's browser, so its calculations were done in advance, on the site's questions.

Sources