Quantization Explained: Q4, Q8 and FP16
Updated
Quantisation is the single biggest lever over what a given machine can run, and understanding it changes which hardware you need to buy.
What is quantisation?
Models are trained at 16-bit precision. Quantisation stores each parameter in fewer bits, which shrinks the model roughly in proportion. The same 70B model needs about 161GB at FP16, about 88GB at Q8 and about 51GB at Q4.
What does it cost in quality?
Less than most people expect down to Q4. The widely used Q4_K_M format keeps quality close enough to the original that the difference is hard to detect on ordinary tasks, while halving the memory requirement against Q8. Below Q4 the losses become noticeable and the additional saving is small, which is why Q4 is the usual stopping point.
Why we calculate at 4.85 bits
Q4_K_M is not a flat four bits per parameter. It uses mixed precision internally, keeping some tensors at higher precision, which averages out around 4.85 bits in practice. Calculating at a clean four would underestimate requirements by around 20 percent, which is the difference between a model fitting and not.
Which should you run?
Q4_K_M for almost everything. Move to Q8 only if you have memory to spare and are doing work where small quality differences matter, such as careful code generation. FP16 is for training and fine-tuning rather than inference, and the memory cost is rarely justified for running a model.
What do the K and M suffixes mean?
Formats such as Q4_K_M use mixed precision internally rather than a single bit width. Certain tensors, typically attention and some feed-forward weights, are kept at higher precision because they are more sensitive to rounding, while the bulk is stored at the lower width. The M denotes a medium-sized variant of that scheme; S and L variants trade size against quality either side of it.
This is why Q4_K_M averages around 4.85 bits per parameter rather than a clean four, and why calculating memory requirements at four bits underestimates them by roughly 20 percent.
Does quantisation affect speed as well as size?
Yes, and favourably. Generation requires reading the weights from memory for every token, so a smaller model means fewer bytes read and faster output. Moving from Q8 to Q4 roughly halves the memory traffic and roughly doubles the token rate on the same hardware. You gain capacity and speed at the same time, which is unusual.
When is Q8 worth it?
When you have memory to spare and the work is sensitive to small errors. Careful code generation, structured extraction where format matters, and long reasoning chains where small mistakes compound are the usual cases. If you have the memory and the model fits, there is no reason not to use it.
What is rarely worth it is choosing a smaller model at Q8 over a larger model at Q4 when both fit. A 70B model at Q4 generally outperforms a 32B model at Q8 despite the coarser quantisation, because parameter count matters more than precision across that range.
What about even smaller formats?
Q3 and Q2 exist and will load a large model on hardware that otherwise could not hold it. The quality cost rises sharply, and it falls hardest on reasoning and instruction following, which are usually the reasons for choosing a large model. If a model only fits at Q2, running a smaller model at Q4 is almost always the better result.
How much memory each level needs
For a 70B model: about 161GB at FP16, about 88GB at Q8, about 68GB at Q6, about 59GB at Q5 and about 51GB at Q4. Those include runtime overhead and an 8k context, and they are the figures this site uses to decide whether a machine can run a model.
The same ladder applies proportionally at other sizes, which is why a 32B model needs about 24GB at Q4 and a 120B model about 86GB.
Bigger model or higher precision?
When both fit, the larger model at lower precision usually wins. A 70B model at Q4 generally outperforms a 32B model at Q8 despite the coarser quantisation, because parameter count contributes more than precision across that range.
This has a direct hardware consequence: memory capacity is more valuable than you might assume from the quantisation table, because it lets you run a fundamentally better model rather than a slightly more precise version of a smaller one.
What to run
Q4_K_M as the default. Q8 when you have memory to spare and the work is precision-sensitive. FP16 only for training or fine-tuning. Below Q4 only when the alternative is not running the model at all, and in that case consider whether a smaller model at Q4 would serve you better.
How we calculate the figures on this page
The memory figures in this guide are produced by the same rule the rest of the site uses, which is why a model's requirement here matches what the compatibility tables show. Weights are the parameter count multiplied by the bits per parameter, divided by eight. Runtime overhead adds about 15 percent. The key-value cache adds roughly 2GB at an 8k context and scales with length from there. The bits per parameter we use are 4.85 for Q4_K_M, 5.7 for Q5, 6.6 for Q6, 8.5 for Q8 and 16 for FP16, and those are averages of mixed-precision schemes rather than nominal widths. Using the nominal four bits for Q4 would understate a 70B model's requirement by around 8GB, which is the difference between a machine fitting and not. One case deserves separate mention. For mixture-of-experts models the full parameter count drives the memory requirement, not the active count, because every expert must be resident even though only some run per token. A 235B model with 22B active needs room for all 235B. Treating the active figure as the requirement is the single most common way people underestimate what a machine needs, and it is why our compatibility calculations always use the full count.
