Home › Guides › GPU memory for LLMs

How to Estimate GPU Memory for an LLM: Weights, KV Cache and Overhead

Updated October 21, 2026 · By the Xingyu Zhitong editors

Almost every hardware question about large language models starts the same way: will it fit? A model that is one gigabyte too large for a card does not run slowly, it simply fails to load, or it loads and then crashes on the first long prompt. The good news is that the dominant terms in GPU memory use are simple multiplications you can do on paper. This guide walks through the estimate in the order you should do it, using the architecture numbers Meta published for Llama 3.1, and points out where the paper estimate stops being reliable and you need to measure.

The four buckets of inference memory

For inference (not training), the memory on each GPU is the sum of four things:

1. Weights. Every parameter stored at some precision. This is fixed once you choose the model and the number format.
2. KV cache. The key and value vectors of every token already processed, kept so attention does not recompute them. It grows with context length and with the number of simultaneous requests. See KV cache memory explained for the full derivation.
3. Activations and temporary buffers. Intermediate tensors during a forward pass. For single-request decoding these are small; during prefill of a long prompt they can be noticeable, which is why serving frameworks cap the number of tokens processed per step.
4. Runtime overhead. The CUDA context, library workspaces, and allocator fragmentation. This is hard to compute and is best treated as a measured margin rather than a formula.

Weights and KV cache are the two you can calculate precisely. The other two you should reserve room for.

Step 1: count the parameters

Model names round the parameter count ("70B"), which is fine for a first estimate, but you can compute it from the architecture. Table 3 of the Llama 3 paper (Dubey et al., 2024) lists the structure of the three sizes:

The paper also gives a vocabulary of 128,000 tokens and a SwiGLU feed-forward block. With these you can rebuild the total. Per layer, with model dimension d, FFN dimension f and head dimension d/heads:

• Query and output projections: 2 × d × d
• Key and value projections: 2 × d × (KV heads × head dim)
• SwiGLU feed-forward (gate, up, down): 3 × d × f

Then add the embedding table and the output head, each vocabulary × d (Llama 3 does not tie them). For the 8B model: head dim = 4,096 / 32 = 128, so the key/value projections are each 4,096 × 1,024. Per layer that is 2 × 4,096² + 2 × 4,096 × 1,024 + 3 × 4,096 × 14,336 = 33.55M + 8.39M + 176.16M = 218.1M parameters. Times 32 layers is 6.98B. Embeddings and head add 2 × 128,000 × 4,096 = 1.05B. Total: about 8.03B, which matches the name. Doing the same arithmetic for 70B gives about 70.55B and for 405B about 405.8B. (Small differences from the official counts come from rounding and from norm weights, which are negligible.)

You will not usually need to do this by hand, but it is worth doing once, because it shows that almost all of the parameters sit in the feed-forward blocks and that the attention key/value projections are small. That is exactly why grouped-query attention, covered below, saves so much cache memory with little effect on weights.

Step 2: multiply by bytes per parameter

Weight memory is parameters × bytes per parameter:

The 4-bit column is a floor, not a promise. Real 4-bit formats store extra scaling factors per block of weights, so the effective size is somewhat above 0.5 bytes per parameter. The llama.cpp llama-bench documentation shows a concrete case: a Qwen2 7B model in Q4_K Medium format with 7.62B parameters occupies 4.36 GiB, which works out to roughly 4.9 bits per weight, not 4. Details on formats are in our quantization guide.

Already at this step you can answer a lot of questions. An 80 GB H100 cannot hold 70B weights in BF16 (141 GB) but can hold them in FP8 (70.6 GB) only just, with almost nothing left for cache. And Meta states directly in the Llama 3 paper that the BF16 405B model does not fit in a single machine with eight H100 GPUs, which is 8 × 80 = 640 GB against 812 GB of weights, so they ran it across 16 GPUs on two machines.

Step 3: add the KV cache for your workload

The per-token cache size is:

KV bytes per token = 2 × layers × KV heads × head dim × bytes per element

The leading 2 is one key and one value vector. For Llama 3.1 8B in BF16: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. A single 8,192-token conversation therefore needs 1 GiB; sixteen of them at once need 16 GiB. For 70B: 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KiB) per token. For 405B: 2 × 126 × 8 × 128 × 2 = 516,096 bytes (504 KiB) per token.

Notice the role of the KV head count. Llama 3 uses 8 key/value heads shared across 32, 64 or 128 query heads. That is grouped-query attention, proposed by Ainslie et al. (2023). If the 70B model had one KV head per query head (64), the cache would be eight times larger, 2.5 MiB per token, which at 8,192 tokens would be 20 GiB per request. Always take the KV head count from the model's config.json (num_key_value_heads), not the attention head count.

Step 4: a margin for activations and overhead

Keep a deliberate buffer. The vLLM documentation describes gpu_memory_utilization as the fraction of GPU memory the engine pre-allocates for its cache, and it explains that preemption (recomputing requests) happens when KV cache space runs out. In practice that means the engine takes your configured fraction of the card, subtracts weights and its own profiled activation peak, and gives the remainder to the cache. If you set the fraction too high you risk out-of-memory errors from other processes on the same GPU; too low and you waste capacity.

A practical planning rule is to keep roughly 10 percent of each card unallocated and to verify with the framework's own startup log, which reports how many KV cache tokens it could allocate. That number is the ground truth for your configuration. Treat your arithmetic as the way to decide which hardware to try, and the log as the way to confirm it.

A complete worked example

Suppose you want to serve Llama 3.1 70B in FP8 to several users with 8,192-token contexts on two 80 GB cards, using tensor parallelism so the weights and cache are split across both GPUs.

1. Weights in FP8: 70.55B × 1 byte = 70.6 GB.
2. Total memory: 2 × 80 = 160 GB. Reserve 10 percent: 144 GB usable.
3. Left for cache and activations: 144 − 70.6 = 73.4 GB. Set aside 5 GB for activations, giving about 68 GB for cache.
4. KV cache in BF16 at 327,680 bytes per token: 68 × 10⁹ / 327,680 ≈ 207,000 tokens.
5. At 8,192 tokens per request: about 25 concurrent requests.

These are example numbers with stated assumptions (the 5 GB activation allowance and the 10 percent reserve are guesses, not measurements). Quantizing the cache to FP8, which vLLM supports, would roughly double the token capacity. Compare this to running BF16: weights alone are 141 GB, leaving just 3 GB of the 144 GB, which is not a workable deployment. This is the typical pattern: precision choice moves you between "impossible" and "comfortable".

Common mistakes

• Using the attention head count instead of the KV head count in the cache formula. This overestimates cache memory by 4 to 8 times for modern models.
• Ignoring concurrency. A cache budget that looks generous for one chat is exhausted by a few dozen simultaneous long sessions.
• Mixing GB and GiB. 80 GB on a datasheet is decimal; a framework may report GiB (2³⁰ bytes). The gap is about 7 percent, enough to decide a borderline case.
• Treating the model name as the parameter count. Some mixture-of-experts models store far more parameters than they use per token. Memory depends on total parameters, speed on active ones; see DeepSeek-V3 and R1 hardware.
• Forgetting multi-GPU overhead. Splitting a model across cards does not divide everything evenly; some tensors are replicated, and each card keeps its own CUDA context and buffers.
• Planning for training with inference numbers. Fine-tuning needs gradients and optimizer states on top; see fine-tuning memory.

Use the calculators to check your arithmetic

Our model-specific calculators apply the same formulas so you can sanity-check a plan: the Llama 3.1 70B, Llama 3.1 405B and Qwen 2.5 7B VRAM calculators take precision and context length and return totals. If the tool and your hand calculation disagree, the usual culprits are the KV head count and the bytes per element.

Once memory fits, the next question is speed, which depends on bandwidth rather than capacity; that is covered in inference speed and tokens per second. For choosing the cards themselves, see the H100, A100, H200 and B200 comparison.

Sources

• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3 and Section 6
• GQA: Training Generalized Multi-Query Transformer Models (Ainslie et al., 2023)
• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
• vLLM documentation: Optimization and tuning
• llama.cpp llama-bench README
• NVIDIA H100 data sheet

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.