How a reply is built · Lesson 7 of 18
Read the prompt, then build the answer
The model reads your prompt to choose the first answer piece, then uses each new piece to choose another.
How it works, step by step
Read the four prompt pieces
Process the prompt in a pass called prefill. Use its scores to choose the first answer token, A.
Four prompt positions saved and one answer token chosen.
Use A to choose B
Feed A through the model. Reuse the saved keys and values to choose token B.
Five positions processed and saved; two answer tokens chosen.
Use B to choose C
Feed B through the model to choose C. Stop at the requested three answer tokens.
Three answer tokens from one prefill plus two decode passes. C need not be processed if the answer stops.
In plain language
Prefill means processing the prompt to get scores for choosing the first answer piece, called a token. Decode then processes the newest token to get scores for choosing the next one.
A way to picture it
First read an order, then work out what to do next using what you have already prepared. The example builds the answer one piece at a time.
A worked example
A 4-token prompt and 3 output tokens need 1 prefill pass and 2 subsequent decode passes in this example. Prefill gives scores for choosing A. Processing A gives scores for B; processing B gives scores for C. C need not be processed if the answer stops there.
Keep in mind
A token can be a whole word, part of a word or punctuation. This example excludes speculative decoding, which tries several future tokens, and assumes no saved prompt prefix or split prompt processing. Time before the first token also includes waiting and other work, not just prefill.
What these words mean
- Token
- A text piece represented by a number. It may be a whole word, part of a word or punctuation.
- Prefill
- Processing the prompt to build reusable working data and scores for the first answer token.
- Decode
- Processing the newest token to get scores for choosing the following token.
- Keys and values
- Saved number lists from already processed positions, reused when the model mixes earlier context.
- Scores
- Numbers used to choose a next token. The model's raw scores are called logits; they are not a promise that an answer is true.
- Example limits
- This example does not reuse an earlier prompt or split prompt processing into chunks. It also leaves out speculative decoding, which tries several possible future tokens.
Prefill helps choose A, then two more passes choose B and C: how many decode passes is that?
Two. Prefill already gives the scores for A. One decode pass processes A to choose B. A second processes B to choose C.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.