Choosing the words · Lesson 12 of 18
Choosing the next piece
Every piece in the vocabulary gets a score, the scores become shares, and a setting decides which piece is taken.
How it works, step by step
Score every candidate
The model produces one raw score, called a logit, for every piece in its vocabulary. Higher means a better fit here.
A score for each candidate piece.
Turn the scores into shares
A calculation turns the scores into shares adding up to 100%. A lower temperature setting makes the leader's share larger.
A share for each candidate piece.
Pick one
Always taking the top share repeats the safest choice. Drawing from the strongest few lets a less likely piece appear instead.
One chosen piece, which becomes the next input.
In plain language
The model does not hand back a word. It produces a score for every piece in its vocabulary, and separate settings decide which piece is actually taken.
A way to picture it
Think of a shortlist with scores beside each name. You can always take the top name, or draw from the strongest few, which changes how predictable the outcome feels.
A worked example
Say three candidate pieces score 2.0, 1.0 and 0.5. Turning those scores into shares gives about 63%, 23% and 14%, and the setting decides whether to take the leader or draw.
Keep in mind
Temperature and similar settings change the shares, not the model. The same prompt can give different answers whenever a draw is allowed, and a large share is not a check that the answer is true.
What these words mean
- Logit
- A raw score the model gives one candidate piece, before the scores become shares.
- Softmax
- The calculation that turns raw scores into shares adding up to 100%.
- Temperature
- A setting that sharpens or flattens the shares before a piece is picked.
- Top-k and top-p
- Settings that keep only the strongest candidates, then draw one of them.
- Not a truth check
- A large share means the piece fits the pattern here. It is not evidence that the answer is correct.
Why can the same prompt give two different answers?
Because a setting allowed a draw between candidates. When the settings allow a draw, a less likely piece can be picked, and one different piece changes everything that follows it. Always taking the top share makes repeats far more likely.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.