Your words become numbers · Lesson 4 of 18
How your text becomes pieces
Before a model sees your text, software splits it into pieces called tokens and swaps each piece for a number.
How it works, step by step
Start with your text
A tokenizer reads your text from the start. It carries a fixed list of pieces it already knows, decided before the model was trained.
Your text, ready to be split.
Cut it into pieces
The text is cut into the longest pieces on that list. A piece can be a whole word, part of a word, a space or a mark.
A row of text pieces, still in order.
Swap each piece for a number
Each piece is looked up in the list and replaced by its position number, called a token ID.
A row of numbers the model can work with.
In plain language
A model does not read letters one at a time the way you do. Software first splits your text into pieces called tokens, and every piece has its own number.
A way to picture it
Think of a cook chopping vegetables before anything is cooked. The chopping rule is fixed in advance, so the same word is usually cut the same way each time.
A worked example
The tokenizer in this app is real. It splits the text you type into pieces and looks up an ID for each one, so a longer word can become more than a single piece.
Keep in mind
A token is neither a word nor a letter. A space or a mark can belong to a piece, another model's tokenizer can split the same text differently, and counting words does not give you a token count.
What these words mean
- Token
- A piece of text the model works with. It may be a whole word, part of a word, a space or punctuation.
- Token ID
- The number that stands for one piece in the tokenizer's fixed list.
- Vocabulary
- The fixed list of pieces a tokenizer knows. Different models carry different lists.
- Tokenizer
- The software that splits text and looks up the numbers. It is not the model, and it does not understand your text.
Is one token always one word?
No, a longer word can be split into several pieces. A short common word is often a single piece, but a longer or unusual one can be cut into several. Spaces and marks can belong to a piece too, so a word count is not a token count.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.