How a reply is built · Lesson 9 of 18
Working on one piece at a time
After attention mixes information between pieces, a second part of the block reworks each piece on its own.
How it works, step by step
Take one piece
Attention has already mixed in what the allowed earlier pieces contribute. This part now looks at one piece by itself.
A single list of numbers to rework.
Expand, then reshape
Our drawing expands the list using learned weights, then applies ReLU, a rule that sets negative values to zero. Qwen2.5 uses a different rule, SiLU with gating.
A longer list reshaped by this example's ReLU rule.
Bring it back to size
Learned weights combine the long list back down to its original length, and the result is added onto the piece it started from.
The same piece, with its numbers reworked.
In plain language
Attention lets the pieces look at each other. A second part of every block then reworks each piece on its own, expanding its numbers and bringing them back.
A way to picture it
Picture a small workshop bench. One item is opened out into many parts, some are set aside, and what is left is reassembled to the size it started at.
A worked example
Say a piece carries 4 numbers. Our simplified drawing expands it to 16, uses ReLU, a rule that sets negative values to zero, then combines it back down to 4.
Keep in mind
The drawing uses ReLU to show an activation rule, which reshapes the numbers between matrix calculations. Qwen2.5 uses SiLU in a gated block called SwiGLU instead. Feed-forward blocks often hold most of a model's stored numbers. The sizes in the example are tiny stand-ins for real ones.
What these words mean
- MLP
- Short for multi-layer perceptron. The part of a block that reworks each piece on its own.
- Feed-forward
- Another name for the same part of the block.
- Activation
- A rule that reshapes numbers so a block can represent more than straight lines. Our drawing uses ReLU; Qwen2.5 uses SiLU in a gated block called SwiGLU.
- Residual
- Adding a part's result back onto what went into it, so earlier information is not lost.
How is this part different from attention?
It works on each piece on its own. Attention is where pieces exchange information with each other. This part reworks each piece separately, which is why every piece can go through it at the same time.
Where this comes from
The linked documentation explains the real hardware and software. Our small number examples help you work through the idea; they do not measure a GPU's speed.