Home › Guides › GPU memory for LLMs

KV Cache Memory Explained: Why Long Chats Eat Your VRAM

Updated October 21, 2026 · By the Xingyu Zhitong editors

When you load a language model, the weights take a fixed amount of memory. Then you start chatting, and memory use keeps climbing, token after token, request after request. The thing that grows is the key-value cache, usually shortened to KV cache. On a busy server it is often the part that decides how many users fit on one GPU; on a long-context workload it can rival the weights themselves. This article explains what is in the cache, derives its size from the model configuration, and shows how recent architectures shrink it.

What the cache actually stores

A transformer generates text one token at a time. To produce each new token, every attention layer compares the new token's query vector against the key vectors of all earlier tokens, then uses the resulting weights to mix their value vectors. The keys and values of earlier tokens never change once computed, so recomputing them at every step would be wasteful. The runtime stores them instead. That store is the KV cache.

Two consequences follow. First, the cache holds one key and one value per token, per layer, per KV head. Second, it is entirely per-sequence: two users chatting with the same model each need their own cache, even though they share the weights. That is why the cache scales with concurrency.

The formula

KV cache bytes = 2 × layers × KV heads × head dim × sequence length × bytes per element × batch size

The 2 counts keys and values. Take layers, KV heads and head dimension from the model's config.json (num_hidden_layers, num_key_value_heads, and head dimension, which is hidden_size / num_attention_heads unless a head_dim field says otherwise). Bytes per element is 2 for FP16/BF16 and 1 for FP8.

The PagedAttention paper (Kwon et al., 2023) gives a classic check of this reasoning with the older 13B-parameter OPT model: 2 (key and value) × 5,120 (hidden size) × 40 (layers) × 2 (bytes in FP16) = 800 KB per token. Since that model could generate up to 2,048 tokens, one request could need up to 1.6 GB of cache. For that model there was one KV head per attention head, so the hidden size stands in for KV heads × head dim.

Worked examples with real configurations

The Llama 3 paper (Table 3) lists 8 key/value heads for all three Llama 3 sizes, and layers of 32, 80 and 126 for 8B, 70B and 405B. The head dimension is the model dimension divided by the number of heads: 4,096/32, 8,192/64 and 16,384/128 all give 128. In BF16:

• Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token.
• Llama 3.1 70B: 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token.
• Llama 3.1 405B: 2 × 126 × 8 × 128 × 2 = 516,096 bytes, or 504 KiB per token.

The Qwen2.5 technical report lists the 7B model with 28 layers and 28 query heads sharing 4 KV heads, and the 72B model with 80 layers and 64 query heads sharing 8 KV heads. Using a head dimension of 128 (the hidden size divided by heads for these models), the 7B cache costs 2 × 28 × 4 × 128 × 2 = 57,344 bytes (56 KiB) per token, and the 72B costs 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KiB) per token, the same as Llama 3.1 70B.

Read the last column carefully: a single 128K-token session on the 8B model needs more cache than the weights of a 4-bit 8B model. On the 70B model, one full-length session needs 40 GiB, half of an 80 GB card's capacity, before any other user arrives.

Why the number of KV heads matters so much

Early transformers used multi-head attention, where every query head had its own key and value head. Multi-query attention uses a single KV head, which shrinks the cache dramatically but can hurt quality. Grouped-query attention (GQA), introduced by Ainslie et al. (2023), is the compromise: a small number of KV heads, each shared by a group of query heads. The paper reports that GQA reaches quality close to multi-head attention with speed comparable to multi-query attention.

For Llama 3.1 70B, the 8 KV heads serve 64 query heads, an 8-to-1 grouping. A hypothetical 70B model with full multi-head attention would need 2 × 80 × 64 × 128 × 2 = 2,621,440 bytes (2.5 MiB) per token, eight times more. At 32K tokens that is 80 GiB for one sequence instead of 10 GiB. GQA is a big reason long contexts are practical on current hardware.

Compressing the cache by design: multi-head latent attention

DeepSeek-V2 and V3 take a different route called multi-head latent attention (MLA). Instead of caching full keys and values, the model caches a compressed latent vector plus a small positional component. The DeepSeek-V3 technical report gives the configuration: 61 layers, 128 attention heads of dimension 128, a KV compression dimension of 512, and a decoupled rotary key dimension of 64. The V2 paper reports that MLA reduces the KV cache by 93.3 percent compared with the earlier DeepSeek 67B model.

Per token and per layer, the cache holds 512 + 64 = 576 elements. In BF16 that is 576 × 2 × 61 = 70,272 bytes (about 68.6 KiB) per token, comparable to a small dense model even though the total model has 671B parameters. For contrast, caching full keys and values for 128 heads of dimension 128 would take 2 × 128 × 128 × 61 × 2 = 3,997,696 bytes (about 3.8 MiB) per token, roughly 57 times more. At 131,072 tokens, the MLA cache is about 8.6 GiB. (The latent-cache layout is as described in the papers; individual inference engines may store it differently, so check your engine's reported capacity.) See DeepSeek-V3 and R1 hardware for the weights side of that deployment.

Cutting cache memory in practice

• Quantize the cache. vLLM supports an FP8 KV cache through the kv_cache_dtype option, halving bytes per element relative to BF16 and roughly doubling token capacity. It can change outputs slightly; the vLLM documentation recommends calibrating scales with a dataset for best accuracy.
• Use paged allocation. Naive servers reserve a contiguous maximum-length buffer per request and waste most of it. PagedAttention allocates the cache in small blocks on demand; the paper reports near-zero waste and 2 to 4 times higher throughput than earlier systems at the same latency.
• Limit context and batch. Setting a realistic maximum context length and maximum concurrent sequences is the simplest control. In vLLM, lowering max_num_seqs or max_num_batched_tokens reduces cache demand, as its tuning guide describes.
• Window or evict. Sliding-window attention and approaches like StreamingLLM keep only recent tokens plus a few initial "sink" tokens, bounding memory at the cost of forgetting the middle of long documents.
• Share prefixes. If many requests start with the same system prompt, prefix caching lets them share those blocks instead of storing copies.

How the cache fills and empties over a request

A request has two phases. During prefill, the model processes the whole prompt in parallel and writes the keys and values for every prompt token into the cache in one go, so a 20,000-token document instantly claims 20,000 tokens of cache. During decoding, each generated token appends one more entry per layer. When the request finishes, its blocks are released. This lifecycle explains two common surprises. The first is that memory use jumps when a long prompt arrives, even before any output appears. The second is that a server that looks idle at the start of the day can run out of cache by mid-afternoon, not because anything leaked, but because the number of simultaneous long sessions grew.

When the cache is full, a scheduler has three options: make new requests wait, preempt a running one and recompute it later (the vLLM tuning guide describes this recompute behaviour and the warning it prints), or reject the request. Each trades latency or availability for memory. Sizing the cache for your peak concurrency, rather than your average, is what avoids all three. A rough planning rule is to multiply the number of simultaneous sessions you expect at peak by their typical total length (prompt plus output) and compare the product with the token capacity your server reports.

Estimating how many users fit

Here is a short example with stated assumptions. You have two 80 GB GPUs serving Llama 3.1 70B in FP8 weights (about 70.6 GB), and you allow 144 GB total after a 10 percent reserve and 5 GB of activation allowance, leaving about 68 GB for cache. With a BF16 cache at 327,680 bytes per token: 68 × 10⁹ / 327,680 ≈ 207,000 tokens. If your typical session holds 4,000 tokens, that is about 51 sessions; if sessions reach 32,768 tokens, about 6. With an FP8 cache you would double both numbers. These figures are illustrative; the server's startup log reports the real token capacity for your configuration.

The same arithmetic is built into our calculators for Llama 3.3 70B and Qwen 2.5 72B. For the bigger picture of how cache combines with weights, read how to estimate GPU memory for an LLM, and for what happens when contexts grow past 100K tokens, see the cost of long context.

Sources

• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023)
• The Llama 3 Herd of Models (Dubey et al., 2024)
• Qwen2.5 Technical Report
• DeepSeek-V3 Technical Report
• DeepSeek-V2: A Strong, Economical, and Efficient MoE Language Model
• vLLM documentation: Quantized KV cache

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.