Look a little closer · Lesson 18 of 18
A big speed number needs a real test
A big hardware speed number does not tell you how quickly a real chat app will answer.
How it works, step by step
Read the small print
A peak describes certain calculations under stated conditions. Check whether it counts one chip or a whole system.
A calculation limit under those conditions, not a promised chat speed.
Try a timed example
In this imaginary test, 120 tokens finish in 6 seconds. Dividing 120 by 6 gives 20 tokens per second.
120 ÷ 6 = 20 output tokens/s for that interval. This is arithmetic, not a device benchmark.
Compare the same kind of test
Check that the models, requests and timing rules are comparable. Look at both total output and each person's wait.
A comparison that says what was tested and what was left out.
In plain language
An advertised peak says how many calculations hardware can handle under stated conditions. A real chat test measures how the model, software and hardware work together on actual requests.
A way to picture it
A car's top speed does not tell you when you will get home. Traffic, the route and stops also matter.
A worked example
Imagine a test that finishes 120 output tokens in 6 seconds: 120 ÷ 6 = 20 tokens/s across that test. This is arithmetic, not a result measured on Spark, H100 or any other listed product.
Keep in mind
Record the model and software versions, how numbers are stored, prompt and answer lengths, and how many requests run together. Say when timing starts and stops, and whether the model was already loaded. A peak may count only certain calculations or skip supported patterns of zero values. Neither Spark's memory size nor its advertised FP4 calculation rate alone tells you chat speed.
What these words mean
- Peak
- An advertised maximum for particular calculations under stated conditions, not a promise for every app.
- FLOPS
- Floating-point operations per second: a count of calculations on a particular format of numbers, not answer tokens.
- Benchmark
- A defined test used to measure how a system performs.
- Test details
- Record model and software versions, number formats, prompt and answer lengths, and requests running together. Say whether the model was already loaded and when timing started and stopped.
- Sparsity
- Some supported calculation methods can skip particular patterns of zero values. A peak using that shortcut is not a promise for all data.
Can the advertised calculation speed alone tell you how fast chat replies will arrive?
No, we need a test of the model and software too. The model, software, amount of work and data movement all matter. Test the setup you plan to use instead of treating a peak as chat speed.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.