Home › Guides › Speed and scaling

Why Your LLM Is Slow: Predicting Tokens per Second from Memory Bandwidth

Updated October 21, 2026 · By the Xingyu Zhitong editors

When people compare GPUs for language models, they usually quote teraflops. For the most common interactive workload, generating text one token at a time, teraflops are almost irrelevant. What decides your tokens per second is how quickly the GPU can read the model from its own memory. Once you see that, speed estimates become a one-line division, and surprising results (a cheap card with fast memory beating a pricier one, a quantized model running twice as fast, a batch of users costing little extra) all make sense. This guide builds the model step by step, with worked numbers and the points where the simple picture breaks.

Two phases with different bottlenecks

A request has a prefill phase and a decode phase, and they stress different resources.

• Prefill processes the whole prompt in parallel. Thousands of tokens share each weight read, so the GPU's arithmetic units are busy. It is compute-bound, and it determines time to first token.
• Decode produces one token per step per request. Each step must read the weights (and the request's KV cache) from memory but does little arithmetic per byte read. It is memory-bandwidth-bound, and it determines the inter-token latency you perceive as typing speed.

The vLLM optimization documentation describes the same split, noting that chunked prefill balances compute-bound prefill work against memory-bound decode work in a batch.

The decode formula

For a single request on a dense model:

tokens per second ≤ memory bandwidth / (weight bytes + KV cache bytes read per token)

Using the published memory bandwidth of the card (NVIDIA lists 3.35 TB/s for the H100 SXM and 4.8 TB/s for the H200) and the model size from our memory estimation method, you get a ceiling. It is an upper bound: real engines reach some fraction of it, because kernels do not achieve peak bandwidth and there are other overheads. Treat 50 to 80 percent as a typical range for well-tuned stacks, and measure.

Weight sizes come from the Llama 3 architecture (about 8.03B and 70.55B parameters). The RTX 4090 bandwidth of roughly 1 TB/s is an approximate figure; look up the exact value for your card. Several lessons are visible. Halving the bytes (quantization) doubles the ceiling. The H200 beats the H100 on the same model by its bandwidth ratio (4.8 / 3.35 = 1.43), not by compute. Adding a second GPU with tensor parallelism raises the combined bandwidth, which is one reason it speeds up decoding, but it also adds communication costs per layer, so the gain is less than 2 times in practice. Compare the splits in tensor versus pipeline parallelism: pipeline parallelism does not raise single-stream speed, since stages work in turn.

Adding the KV cache to the bytes

Each step also reads the request's cache. For Llama 3.1 8B in BF16 that is 128 KiB per token, so at 8,192 tokens of context the cache is 1 GiB (about 1.07 GB), adding roughly 7 percent to the 16.1 GB of weights. At 131,072 tokens it is 16 GiB, which doubles the bytes. That is why speed declines as conversations grow; see long-context cost and KV cache memory explained.

Batching: how a server gets thousands of tokens per second

The weights are read once per step no matter how many requests are in the batch, so a batch of B requests produces B tokens for almost the same weight traffic. Only the cache traffic grows per request. Worked example for Llama 3.1 8B in BF16 on an H100 SXM (3.35 TB/s), with 32 requests each at 8,192 tokens of context:

• Weights read per step: 16.1 GB.
• Cache read per step: 32 × 1.07 GB ≈ 34.4 GB.
• Total: 50.5 GB, so the step time is at least 50.5 / 3,350 ≈ 15.1 ms.
• Each user sees about 66 tokens per second; the server produces about 32 × 66 ≈ 2,100 tokens per second in total.

With a single request the same card shows a ceiling near 200 tokens per second per user, so the 32 users each pay a threefold slower stream but the server delivers about ten times the total output. The arithmetic per step is tiny (32 tokens × about 16 GFLOP each is half a teraflop), so compute is not yet limiting. Eventually, as batches grow, compute or the available cache capacity takes over. This is the economic logic of serving: batch as much as latency targets allow. It is also why cache capacity (memory) and not just bandwidth limits throughput. These are illustrative calculations, not benchmarks; the PagedAttention paper reports 2 to 4 times higher throughput for vLLM than earlier systems at the same latency, largely by wasting less cache memory and so fitting larger batches.

Prefill: estimate from operations

A dense model needs about 2 × parameters floating-point operations per token. That is about 16 GFLOP per token for the 8B model and 141 GFLOP for the 70B model. Suppose, as an assumption for illustration, that your stack sustains 400 TFLOPS on prompt processing. Then the 8B model processes about 400 × 10¹² / 16 × 10⁹ ≈ 25,000 prompt tokens per second, so an 8,192-token prompt takes about 0.33 seconds. The 70B model manages about 2,800 tokens per second, so the same prompt takes about 2.9 seconds. Attention adds extra, growing quadratically with length. For hardware choice, prefill speed tracks compute throughput, which is why data-center GPUs with strong low-precision math shine for long prompts. To get actual numbers, use the llama-bench tool from llama.cpp, whose README documents separate prompt-processing (pp) and token-generation (tg) tests; the numbers differ by an order of magnitude or more.

Mixture-of-experts models

For an MoE model, a single stream reads only the active experts per token, but large batches touch most experts. So single-stream speed follows active parameters (37B of 671B for DeepSeek-V3, per its technical report) while memory capacity follows total parameters, and real throughput is dominated by communication between GPUs. See DeepSeek-V3 and R1 hardware.

Ways to beat the bandwidth limit

• Quantization. Fewer bytes per weight means more tokens per second, up to the efficiency of the kernels; see quantization formats.
• Speculative decoding. A small draft model proposes several tokens, and the big model verifies them in one parallel pass. Leviathan et al. (2022) show it can produce exact samples from the large model's distribution faster. The vLLM documentation says it reduces inter-token latency for medium-to-low-QPS, memory-bound workloads and lists methods including draft models, EAGLE, n-gram lookup and multi-token prediction. Gains depend on how often the draft is right and are smaller when the server is already busy with large batches.
• More bandwidth. Tensor parallelism across GPUs, or simply a card with faster memory, raises the ceiling.
• Smaller or sparser models. If a 7B model satisfies your task, it will be several times faster than a 70B one at the same precision; see 7B and 8B models on one GPU.

Why advertised and measured speeds disagree

When two sources quote different tokens-per-second figures for the same GPU and model, check four things before assuming one is wrong. First, which quantity is reported: single-stream generation speed, total throughput across a batch, or prompt-processing speed. They can differ by a factor of ten or more. Second, the context length and batch size used; a short prompt with a batch of one is the most flattering case. Third, the precision and file format, since an 8-bit and a 4-bit model of the same name differ by about twofold in bytes read. Fourth, the software version and settings, including whether flash attention, graph capture or speculative decoding were enabled. A fair comparison fixes all four. If a vendor figure is a projection or uses sparsity-based peak numbers, treat it as a ceiling on arithmetic rather than a prediction of your text-generation speed, which will track memory bandwidth instead.

How to measure your own system

1. Record three numbers separately: time to first token, inter-token latency (or tokens per second after the first), and total throughput under concurrent load.
2. Use realistic prompt lengths and output lengths. Benchmarks with 128-token prompts hide prefill and cache costs.
3. Compare against your ceiling. If you reach well under half of it, check that all layers are on the GPU, that you are using a fast attention kernel, and that the CPU is not the bottleneck (CPU and GPU bottlenecks).
4. Compare engines on the same model and settings; our vLLM versus llama.cpp comparison covers differences.

To translate memory fit into a hardware plan before benchmarking, use the Qwen 2.5 7B or Llama 3.1 70B calculators, and for a desktop build check power headroom with the RTX 4090 PSU calculator.

Sources

• NVIDIA H100 data sheet (memory bandwidth)
• NVIDIA H200 data sheet (memory bandwidth)
• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)
• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
• vLLM documentation: Speculative decoding
• vLLM documentation: Optimization and tuning
• llama.cpp llama-bench README

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.