Memory Bandwidth and LLM Speed
Updated
If memory capacity decides what you can run, memory bandwidth decides how pleasant it is to use. The second number gets far less attention than it deserves.
Why does bandwidth set the speed?
Generating one token requires reading every active parameter of the model from memory. A 70B model at Q4 means reading roughly 42GB per token. That makes generation a memory-bound problem rather than a compute-bound one, and throughput scales close to linearly with bandwidth.
What do the numbers mean in practice?
Divide bandwidth by model size for a rough ceiling on tokens per second. A 42GB model on a machine reading 256 GB/s gives a theoretical ceiling around six tokens per second; the same model at 819 GB/s gives around nineteen. Real figures come in below those ceilings, but the ratio between machines holds.
Why we do not publish tokens per second
Because we have not measured them. A quoted rate depends on the runtime, the quantisation, the context length and the thermal state of the machine, and a figure without those details is worse than no figure. Where a system has no benchmark on record, this site says performance not yet benchmarked rather than inventing one.
How should bandwidth change your decision?
Use it to break ties. When two machines both hold your target model, the one with higher bandwidth is meaningfully better and the difference is immediately visible. When only one holds the model at all, bandwidth is irrelevant, because capacity decides first.
Does bandwidth matter for prompt processing too?
Less. Reading a long prompt is compute-bound rather than memory-bound, because the model processes many tokens in parallel rather than reading all its weights per token. A machine with high compute and modest bandwidth can ingest a long document quickly and then generate the reply slowly, which is a combination that surprises people.
If your work involves long documents and short answers, weight compute more heavily than this page's general advice suggests. If it involves short prompts and long answers, bandwidth is almost the whole story.
Why do real figures fall below the theoretical ceiling?
Several reasons compound. Memory access is not perfectly efficient, the runtime adds its own overhead, the key-value cache has to be read alongside the weights, and thermal limits reduce clocks under sustained load. In practice, expect somewhere between 50 and 80 percent of the theoretical figure depending on the machine and the runtime.
The ratio between two machines survives all of that, which is why bandwidth remains the right number for comparing them even though it overstates both.
How does quantisation interact with bandwidth?
Directly and helpfully. A smaller model means fewer bytes read per token, so the same machine generates faster. Moving a 70B model from Q8 to Q4 roughly halves the bytes read and roughly doubles the token rate, which is a second reason Q4 is the usual choice beyond the memory saving.
What should you take from this?
Use capacity to decide what you can run, and bandwidth to decide how pleasant it will be. When two machines both hold your model, bandwidth is the deciding number and the difference is immediately visible. When only one holds it, bandwidth is irrelevant.
What the numbers look like across this catalogue
The spread is wider than most people expect. A top discrete card reads at 1,792 GB/s. A professional card matches it. A high-memory Mac reaches 1,200 GB/s at the top of the range and 460 GB/s in the middle. Ryzen AI Max sits at 256 GB/s and GB10 at 273 GB/s. Shared-memory laptops land between 100 and 140 GB/s.
That is a factor of more than ten between the fastest and slowest machines on this site, and it shows up directly in how long you wait for an answer.
Why the biggest-memory machines are not the fastest
Capacity and bandwidth are bought differently. Capacity comes from fitting more memory chips; bandwidth comes from widening the path to them. Unified designs optimise for capacity within a power and size budget, and discrete cards optimise for bandwidth within a capacity budget.
This is why a GB10 machine holds a far larger model than a graphics card while generating more slowly, and why that is an engineering trade rather than a flaw in either product.
Using this when you shop
Both numbers appear on every system page and in the leaderboard. Filter on capacity first, because it decides what runs. Sort by bandwidth second, because it decides how the machine feels to use. Any comparison that quotes only one of the two is incomplete.
How we calculate the figures on this page
Bandwidth figures on this site come from vendor specifications and are stored per configuration rather than per product line, because the same chip name can ship at different widths. Apple's M5 Max is the clearest case: 460 GB/s with the 32-core graphics processor and 614 GB/s with the 40-core part, and since the 40-core variant is what unlocks the larger memory options, the two figures attach to different configurations of the same machine. Storing one number for both would misdescribe whichever you were looking at. We deliberately do not convert bandwidth into a predicted tokens per second figure. The relationship is close to linear in theory and messier in practice, because runtime overhead, key-value cache reads and thermal limits all intervene, and the proportion they consume varies by machine. Publishing a derived rate would give the appearance of measurement without the substance of it. What the bandwidth column is genuinely good for is comparison: between two machines that both hold your model, the one with higher bandwidth will be faster in roughly that ratio, and that is a reliable basis for choosing. Capacity first, bandwidth second, and nothing else on the specification sheet comes close to either for predicting how a machine will feel to use.
