Your words become numbers · Lesson 5 of 18
Numbers that carry meaning
Each token ID is swapped for a learned list of numbers, so pieces used in similar ways end up with similar lists.
How it works, step by step
Start with the IDs
An ID is only a label. Being next to another number on the list says nothing about what a piece means.
Four labels, waiting to be looked up.
Look up a list for each
Each ID picks one row out of a table the model learned during training. That row is a list of numbers.
One list of numbers for each piece.
Similar pieces sit close together
Pieces used in similar ways were given similar lists during training, so the model can compare pieces by their numbers alone.
A grid of numbers standing for your prompt.
In plain language
A token ID is only a label, in the way a seat number is only a label. The model swaps each ID for a list of numbers that places the piece near others used in similar ways.
A way to picture it
Think of a map of a city rather than a list of addresses. Two addresses can look similar and be far apart, while places drawn close together on the map tend to be alike.
A worked example
Say each piece becomes a list of 4 numbers. Four prompt pieces then become 4 lists of 4 numbers, which is 16 numbers standing for the whole prompt.
Keep in mind
The lists are learned during training, so they come from the model rather than from a dictionary. Real models use far longer lists than four numbers, and a single position in a list has no plain-English name.
What these words mean
- Embedding
- The list of numbers that stands for one piece of text.
- Vector
- Another word for an ordered list of numbers.
- Learned
- Set during training rather than written down by a person.
- Dimension
- One position in the list. Real models use hundreds or thousands of positions, not four.
What is the model actually working with after this step?
Lists of numbers, one for each piece. From here on the model handles numbers only. Your letters are not carried through the blocks; text is rebuilt at the end from the pieces the model chooses.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.