Choosing the words · Lesson 13 of 18
Numbers become words again
Each chosen piece is turned back into characters and sent on straight away, so an answer appears while it is still being made.
How it works, step by step
Turn a piece back into characters
The same fixed list used to split your text is read the other way, turning the ID back into the characters it stands for.
A short run of characters.
Send it on and keep going
Those characters are added to what you can already see, and the piece is fed back in so the next one can be chosen.
A reply that grows piece by piece.
Stop at the end
Generation ends when the model picks a stop signal or the requested number of pieces is reached, whichever comes first.
A finished reply, already fully delivered.
In plain language
The answer leaves the model as a run of chosen pieces, not as finished text. Each piece is turned back into characters and sent to you as soon as it is picked.
A way to picture it
Think of a reply being read aloud while it is still being written. You hear the opening long before the closing sentence exists.
A worked example
Say the model picks three pieces in order. Each is turned back into its characters and added to what you can already see, so the reply grows until a stop is reached.
Keep in mind
Text that has appeared is already chosen and will not be revised. A piece can be part of a word, so the reply can grow in fragments, and generation ends on a stop signal or the requested limit.
What these words mean
- Detokenise
- Turning chosen pieces back into readable characters.
- Streaming
- Sending each piece on as it is chosen instead of waiting for the whole answer.
- Stop signal
- A special piece the model can choose to end its answer.
- Time to first token
- The wait from a stated starting point until the first piece reaches you.
Why does a reply appear a bit at a time?
Each piece is sent on as soon as it is chosen. The answer is built one piece at a time, and each piece can be sent the moment it is picked. That is why you see an opening long before the reply is finished.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.