How a reply is built · Lesson 9 of 18
Working on one piece at a time
After attention mixes information between pieces, a second part of the block reworks each piece on its own.
How it works, step by step
Take one piece
Attention has already mixed in what the allowed earlier pieces contribute. This part now looks at one piece by itself.
A single list of numbers to rework.
Expand, then simplify
The list is expanded into a much longer one using learned weights. A simple rule then keeps some values and flattens the rest to zero.
A longer list with some values switched off.
Bring it back to size
Learned weights combine the long list back down to its original length, and the result is added onto the piece it started from.
The same piece, with its numbers reworked.
In plain language
Attention lets the pieces look at each other. A second part of every block then reworks each piece on its own, expanding its numbers and bringing them back.
A way to picture it
Picture a small workshop bench. One item is opened out into many parts, some are set aside, and what is left is reassembled to the size it started at.
A worked example
Say a piece carries 4 numbers. This part expands it to 16, applies a simple rule that keeps some values and flattens the rest to zero, then combines it back down to 4.
Keep in mind
The expanding and shrinking steps use learned weights, so most of a model's stored numbers sit here rather than in attention. The sizes in the example are tiny stand-ins for real ones.
What these words mean
- MLP
- Short for multi-layer perceptron. The part of a block that reworks each piece on its own.
- Feed-forward
- Another name for the same part of the block.
- Activation
- The simple rule that keeps some values and flattens others, so a block can represent more than straight lines.
- Residual
- Adding a part's result back onto what went into it, so earlier information is not lost.
How is this part different from attention?
It works on each piece on its own. Attention is where pieces exchange information with each other. This part reworks each piece separately, which is why every piece can go through it at the same time.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.