Look a little closer · Lesson 15 of 18
Using less space for each number
Quantization saves space by using fewer bits, the tiny units of computer data, to store each number.
How it works, step by step
Count the starting space
Multiply 8 by 16 to count the bits. Divide by 8 because a byte holds 8 bits.
8 × 16 ÷ 8 = 16 bytes for the numbers.
Use fewer bits
Use four bits instead of sixteen for each number. Some values may change because fewer numbers can be represented exactly.
8 × 4 ÷ 8 = 4 raw bytes, before extra supporting data.
Try it and check
Check that the software supports it, its answers are good enough and its speed meets your needs. Count extra memory too.
Less space for the numbers, not a promise of faster replies.
In plain language
Quantization stores numbers using fewer bits, the small units that make up computer data. This can save memory, but some numbers change and the model's answers may change too.
A way to picture it
Rounding a map location uses fewer digits. Check that the rounded location still gets you close enough to the right place.
A worked example
8 numbers at 16 bits each need 16 bytes; at 4 bits each they need 4 bytes. That is 4× less space for the numbers alone. Extra data used to interpret and arrange them still needs room.
Keep in mind
Extra costs include scales, numbers that help interpret stored values; metadata, information about their format; and padding, unused space for arranging data. Four-bit weights do not guarantee 4× faster replies or make the KV cache four-bit. INT4, FP4 and NVFP4 are different ways to store numbers, not interchangeable labels.
What these words mean
- Bit width
- How many 0-or-1 digits are used to store each value.
- Precision
- How closely the stored numbers can represent the numbers you started with.
- Scale
- An extra number used by many formats to help turn a stored value back into a useful number.
- Extra storage
- Format information, scales and empty spaces used to arrange data all need room. The saved conversation data and temporary work also use memory.
What does changing eight numbers from 16 bits to 4 bits tell us for sure?
The numbers alone use 4 bytes instead of 16. The numbers alone take 4 bytes instead of 16. Extra data still needs room, and speed and answer quality need a real test.
Where this comes from
This explanation is checked against primary documentation. The small arithmetic examples are ours and are not hardware measurements or vendor benchmarks.