Look a little closer · Lesson 17 of 18
When GPUs need to share results
GPUs sharing one model's work may need to send partial answers to one another.
How it works, step by step
Choose who takes part
Use only the GPUs assigned to this model. A request on one GPU does not need a GPU-to-GPU exchange.
One GPU working alone, or a chosen team of GPUs.
Add the partial answers
Add matching numbers: 2 + 5 = 7 and 3 + 7 = 10.
The combined answer is [7, 10].
Give both GPUs the answer
Give the combined sum to both GPUs. This kind of shared calculation is called an all-reduce.
GPU A and GPU B both have [7, 10].
In plain language
When a model's work is split across GPUs, those GPUs may need to exchange results. A model running on one GPU does not require that exchange with another GPU.
A way to picture it
Two people solving parts of the same puzzle may need to share their answers. A person working alone does not need to pass an answer to everyone else in the building.
A worked example
GPU A has [2, 3] and GPU B has [5, 7]. Add matching numbers and give [7, 10] to both GPUs. This shared sum is called an all-reduce; the example shows its result, not the exact messages or their speed.
Keep in mind
PCIe connects many kinds of computer parts. NVLink connects supported processors, while NVLink-C2C is a different chip-to-chip link used in certain CPU/GPU designs. Ethernet and InfiniBand are network technologies. Check which connection exists and whether its stated speed covers one direction, one GPU or a whole rack.
What these words mean
- PCIe
- A connection used by many kinds of computer parts.
- NVLink
- NVIDIA's connection technology for supported processors. Not every GPU has it.
- All-reduce
- Combine results from a chosen group and give the combined result to everyone in that group. Our example shows the result, not the exact messages or timing.
- NVLink-C2C
- A different chip-to-chip link in certain CPU/GPU designs. It helps those chips keep a consistent view of shared data; it is not the rack's GPU-to-GPU network.
A model uses one GPU in a rack of 72: must its answer travel between GPUs?
No, the model is using only one GPU. GPUs present in a rack are not automatically part of a request. A one-GPU model does not need a cross-GPU shared calculation.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.