Home › Guides › GPU memory for LLMs

What a 128K Context Window Really Costs in Memory and Compute

Updated October 21, 2026 · By the Xingyu Zhitong editors

Model cards advertise context windows of 128K tokens as if they were a free feature. They are not. A longer window changes three separate costs: the memory that stores the conversation, the arithmetic needed to read a long prompt, and the speed of every token generated afterwards. Each grows differently, and confusing them leads to hardware that technically loads the model but cannot actually serve long requests. This guide separates the three, puts numbers on each for the Llama 3.1 family, and describes the methods used to keep them in check.

Cost one: memory grows linearly with length

The key-value cache stores two vectors per token per layer per KV head, so its size is proportional to the number of tokens: 2 × layers × KV heads × head dim × bytes × tokens. For Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128; Table 3 of the Llama 3 paper) that is 131,072 bytes per token in BF16, so a full 131,072-token window holds 16 GiB. For 70B (80 layers, 8 KV heads) it is 327,680 bytes per token and 40 GiB for the full window. The detailed derivation, and the table comparing models, are in KV cache memory explained.

The key point here is how this interacts with concurrency. The cache is per request. If four users each submit 128K-token documents to a 70B model, the cache alone is 4 × 40 = 160 GiB, more than the 141 GB of BF16 weights and more than the entire memory of two 80 GB GPUs. A server configured for "128K context" with ample memory for 128K at one user may have memory for only a handful at once.

Cost two: prefill compute grows faster than linearly

When a long prompt arrives, the model processes all its tokens before producing the first output token. Two kinds of arithmetic are involved:

• Linear layers (attention projections and feed-forward): about 2 × parameters floating-point operations per token, so total work proportional to length n.
• Attention scores: each token is compared with all earlier tokens, so total work is proportional to n². Per layer, computing the score matrix and applying it to values costs about 4 × n² × d operations, where d is the model dimension (counting the full square; causal masking roughly halves it).

Plug in Llama 3.1 8B (d = 4,096, 32 layers, about 8.03B parameters). At n = 8,192: linear work is 2 × 8.03 × 10⁹ × 8,192 ≈ 1.3 × 10¹⁴ operations, and attention-score work is 4 × 8,192² × 4,096 × 32 ≈ 3.5 × 10¹³, about 27 percent as much. At n = 131,072: linear work is about 2.1 × 10¹⁵ and attention-score work is about 9.0 × 10¹⁵, over four times larger (about twice as large after the causal-mask halving). So at 128K tokens the dominant cost has flipped from the feed-forward layers to attention. These are example calculations from the architecture, not benchmarks, but they explain why time to first token for very long prompts is far worse than a linear extrapolation from short ones.

Cost three: every generated token reads the whole cache

Decoding is limited by memory traffic. For each new token the GPU reads all the weights and the entire KV cache of that request. At short contexts the weights dominate; at long contexts the cache catches up. For Llama 3.1 8B in BF16, weights are 16.1 GB and a 128K-token cache is 17.2 GB (16 GiB), so each token reads about 33 GB instead of 16 GB, and the bandwidth-bound speed ceiling roughly halves. For 70B in FP8 (70.6 GB of weights) against a 40 GiB (43 GB) cache, the bytes per token rise from 70.6 to 113.6 GB, about 61 percent more, so the ceiling falls by about 38 percent. This is why long conversations feel slower as they grow, even when memory is available. More on the bandwidth model in tokens per second explained.

The attention matrix problem and FlashAttention

A naive implementation would materialize the n × n score matrix. For one head at n = 131,072 in 16-bit that is 131,072² × 2 bytes ≈ 34 GB, per head and per layer, which is obviously impossible. FlashAttention (Dao et al., 2022) computes exact attention in tiles, never storing the full matrix, so memory for the attention step grows linearly with n rather than quadratically, and it reduces traffic between GPU main memory and on-chip memory. Every modern serving stack relies on it or an equivalent. It does not reduce the arithmetic, which remains quadratic; it removes the memory wall.

Techniques that reduce the bill

Change the architecture

Grouped-query attention, used by all three Llama 3 sizes with 8 KV heads, cuts cache per token by the ratio of query heads to KV heads compared with full multi-head attention (4 times for 8B, 8 times for 70B, 16 times for 405B). DeepSeek's multi-head latent attention goes further, caching 576 elements per layer; with 61 layers that is 70,272 bytes per token in BF16, about 8.6 GiB for a full 128K window despite the model's 671B parameters.

Compress the cache

Storing the cache in FP8 instead of BF16 halves it. vLLM documents an FP8 KV cache option with optional calibration; check output quality on your tasks, since errors in cached keys affect every later token.

Schedule long prompts sensibly

The vLLM documentation describes chunked prefill, which splits large prompts into smaller chunks and batches them with decode requests so a single long prompt does not stall everyone else. It is enabled by default in the V1 engine whenever possible. Setting a maximum number of batched tokens bounds the activation memory used by a prefill step.

Reuse and limit

If many requests share the same long prefix, such as a system prompt or a document that several users ask about, prefix caching lets them share cache blocks instead of duplicating them. If you can tolerate forgetting the middle of the history, windowed attention keeps only recent tokens; the StreamingLLM paper (Xiao et al., 2023) shows that keeping a few initial "attention sink" tokens along with a recent window lets models generate over very long streams with bounded memory, though it does not give the model access to the discarded text.

Does the model actually use long context well?

A window being supported does not mean every position is used equally well. The paper "Lost in the Middle" (Liu et al., 2023) found that language models often use information at the beginning and end of long inputs better than information in the middle. Llama 3's authors describe extending the context window to 128K tokens in a late continued-pretraining stage, with RoPE base frequency raised to 500,000. Before paying for long-context hardware, test your own task at your real lengths. Often a retrieval step that selects the relevant 5,000 tokens beats pasting tokens, in cost as well as accuracy.

Long outputs count too

Context length is the sum of the prompt and everything the model generates. A reasoning model that writes thousands of tokens of intermediate thinking before its answer fills the cache as surely as a long document does, and it holds that memory for the whole generation. If your workload involves such models, plan the cache against prompt plus expected output, and cap the maximum number of generated tokens in the server settings so a runaway generation cannot consume the budget. The same applies to multi-turn chat: unless the application trims or summarizes old turns, each reply is appended to a history that is resent and re-cached on every turn, so a conversation that starts small can quietly grow into a long-context request after a few dozen exchanges.

A planning example

You plan to serve Llama 3.1 70B in FP8 on two 80 GB GPUs. Weights are 70.6 GB. With a 10 percent reserve and a 5 GB activation allowance (assumptions), about 68 GB remains for cache. With a BF16 cache at 327,680 bytes per token, that is about 207,000 tokens in total. If you allow 128K-token requests, a single one uses 131,072 tokens, so the node can hold about 1.6 of them: one long request plus a few short ones, not a crowd. Switching to an FP8 cache doubles the capacity to about 3 long requests. Moving to eight H200s (1,128 GB) changes the picture entirely. Use the Llama 3.1 70B and Qwen 2.5 72B calculators to vary the context and see where memory runs out, and read our memory estimation guide for the full budget method.

Checklist before promising long context

1. Compute cache per token and multiply by the longest context times the number of concurrent long requests you actually expect.
2. Estimate time to first token with the quadratic attention term in mind, and test with real 50K to 100K token prompts.
3. Measure generation speed at length, not only on short prompts.
4. Decide whether FP8 cache, prefix caching or retrieval can lower the requirement.
5. Set explicit limits for maximum prompt size and concurrency so one user cannot exhaust the cache.

Sources

• The Llama 3 Herd of Models (Dubey et al., 2024)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022)
• Efficient Streaming Language Models with Attention Sinks (Xiao et al., 2023)
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
• DeepSeek-V3 Technical Report
• vLLM documentation: Optimization and tuning (chunked prefill, preemption)
• vLLM documentation: Quantized KV cache

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.