Look a little closer · Lesson 16 of 18
More replies or a shorter wait?
A server can produce more total output while each person still waits longer for a reply.
How it works, step by step
Group some requests
The serving software chooses work it can process together. This group of work is called a batch.
Several requests can share a step without sharing their answers.
Count all the output
Add the output from all four people: 4 × 10 = 40.
40 output tokens in total. We need elapsed time to calculate tokens per second.
Check each person's wait
Measure each person's wait for the first piece, then the gaps between later pieces.
Total output and each person's waiting time are separate measures.
In plain language
A batch groups work from several requests so the GPU can process it together. This can increase the total work finished, but each person may still wait longer for an answer.
A way to picture it
A bus carries more people per trip than a car. That does not mean every passenger reaches home sooner, especially if they wait for the bus to leave.
A worked example
4 requests producing 10 tokens each give 4 × 10 = 40 output tokens in total. That is not one person's output speed. We also need the elapsed time to calculate tokens per second.
Keep in mind
Throughput means total completed work per unit of time. Latency means a particular wait, such as the delay before the first token. Compare both using similar requests and the same number of people. New requests can join later batches; they do not all have to start together.
What these words mean
- Batch
- Work from several requests processed together in one model step.
- Throughput
- Total completed work divided by the time it took.
- Latency
- How long a particular part of a request takes, such as waiting for the first answer piece.
- Time to first token
- The wait from a stated starting point until the first answer token. Say whether this includes time waiting in a queue.
- Concurrency
- How many requests are in progress at once. They do not all have to be in the same batch.
More total tokens finish each second: does every person get a faster reply?
No, we must also check each person's wait. The server may handle more people together while each waits longer. Count the total output and measure individual waits separately.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.