How a reply is built · Lesson 8 of 18
Mixing information from earlier text
Attention mixes information from allowed text positions, giving each position a calculated share in the result.
How it works, step by step
Score the allowed positions
Compare the current position's query with keys from allowed positions. Do not use later text positions.
Scores for how strongly each allowed position can contribute.
Give each value a share
Turn the scores into shares that add up to 100%. Our example supplies the shares 25% and 75%.
The shares are [0.25, 0.75].
Mix the two numbers
Take a quarter of 2 and three quarters of 10. Add them: 0.25 × 2 + 0.75 × 10.
The mixed result is 8.
In plain language
Attention decides how strongly allowed text positions contribute to a new mix of information. In this kind of language model, a position can use itself and earlier positions, but not later ones.
A way to picture it
Mix two ingredients in different amounts. Changing their shares changes the result. Here the ingredients are numbers, not proof that the model understands a sentence.
A worked example
Give the number 2 a 25% share and the number 10 a 75% share. The mix is 0.25 × 2 + 0.75 × 10 = 8. We supply those shares for this small example; a real model calculates them.
Keep in mind
The model compares number lists called queries and keys to get scores. Softmax turns those scores into shares that add to 100%, then those shares mix number lists called values. Our example shows only the mixing. A KV cache avoids repeating old key/value calculations, not all future attention work.
What these words mean
- Query
- A list of numbers calculated from the current position, then compared with keys to score other allowed positions.
- Key
- A list of numbers calculated from a position, used in comparisons with queries.
- Value
- A list of numbers from a position that contributes to the final mixture.
- Softmax
- A calculation that turns scores into shares adding to 100%. We supply its result in this example rather than calculate the full attention process.
- Attention weight
- A share used in the mixture. These shares are calculated for the input, unlike the model's fixed learned weights.
- Causal mask
- A rule that lets a text position use itself and earlier positions, but not later ones.
What is 25% of 2 plus 75% of 10?
8. 25% of 2 is 0.5, and 75% of 10 is 7.5. Together they make 8. This shows the mixing step, not every part of attention.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.