Choosing the words · Lesson 13 of 18
Numbers become words again
Streaming sends readable text chunks while later tokens are still being generated. A chunk can contain several tokens.
How it works, step by step
Turn IDs back into text
The tokenizer converts IDs back into text. Some tokens contain only part of a character's encoded bytes, so it may wait for more tokens before text is readable.
A readable text chunk, possibly from several tokens.
Send it on and keep going
Our drawing sends each readable piece separately. Real servers can group several tokens into one chunk while the model keeps choosing later tokens.
A reply that grows chunk by chunk.
Stop at the end
Generation ends when the model picks a stop signal or the requested number of pieces is reached, whichever comes first.
A finished reply, already fully delivered.
In plain language
The answer leaves the model as chosen token IDs, which software turns back into text. With streaming, readable chunks can reach you while later tokens are still being generated.
A way to picture it
Think of a reply being read aloud while it is still being written. You hear the opening long before the closing sentence exists.
A worked example
Our drawing sends three readable pieces in order. Each adds to what you can already see, so the reply grows until a stop is reached. A real text chunk can contain several tokens.
Keep in mind
Some tokens hold only part of a character's encoded bytes, so the text converter may wait for more tokens. Servers can also group several tokens into one chunk. Once a chunk appears in this example, it will not be revised. Generation ends on a stop signal or the requested limit.
What these words mean
- Detokenise
- Turning chosen token IDs back into readable text. Some tokens need to be combined before their text is readable.
- Streaming
- Sending readable text chunks while generation continues. A chunk can contain several tokens.
- Stop signal
- A special piece the model can choose to end its answer.
- Time to first token
- The wait from a stated starting point until the first answer token. Text buffering and delivery can add time before you see it.
Why can a streamed reply appear before it is finished?
Ready text chunks can be sent while later tokens are generated. The server can send readable text while the model keeps generating. It may combine tokens or wait for complete characters, so one visible chunk is not necessarily one token.
Where this comes from
The linked documentation explains the real hardware and software. Our small number examples help you work through the idea; they do not measure a GPU's speed.