Your words become numbers · Lesson 6 of 18
The model's knowledge and its working notes
The model reuses its learned numbers while keeping separate working notes for the text it is processing.
How it works, step by step
Get the model ready
Load the weights before serving requests. Reuse them without training the model again.
Weights ready to reuse across requests.
Save work from the prompt
The model computes useful numbers called keys and values. It saves them for the positions it has processed.
A KV cache: saved keys and values, not plain-text notes.
Add the next processed piece
If the answer continues, feed that token through the model. Then save its new keys and values.
More saved work, with the same weights. Choosing a token alone does not compute its keys and values.
In plain language
Weights are numbers learned during training, and they normally stay fixed while the model answers. A separate store called the KV cache keeps keys and values, numbers the model can reuse from text it has already processed.
A way to picture it
Weights are a recipe book. The KV cache is a notebook for the order being made. Another order can use the same recipe but may need its own notes.
A worked example
A small cache has 2 layers of model work, 2 KV heads or groups per layer, 4 numbers per head, 3 text tokens and 2 bytes per number. Counting both keys and values gives 2 × 2 × 2 × 4 × 3 × 2 = 192 bytes, before storage-management costs.
Keep in mind
Count KV heads, the groups storing keys and values, rather than query heads, which use them. Some models let several query heads share one KV head. Reusing earlier text, keeping less history or storing smaller numbers can change cache size. Choosing a token alone does not compute its keys and values.
What these words mean
- Weight
- A number learned during training. It normally stays fixed while the model answers a request.
- Token
- A numbered piece of text. It can be a word, part of a word or punctuation.
- Keys and values
- Number lists used by attention to mix information from text positions.
- KV cache
- Saved keys and values from processed positions. It is working data, not a plain-text chat history.
- Cache size
- Longer text can need more saved data. Keeping only recent positions, sharing saved prompt work or using smaller numbers can change how much space is needed.
What usually needs more space as more text is processed?
The saved keys and values for that text. More processed text can need more saved keys and values. The model's learned weights stay the same while it answers.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.