Home › Guides › GPU memory for LLMs

Which GPU Do You Need for a 7B or 8B Language Model?

Updated October 21, 2026 · By the Xingyu Zhitong editors

Models in the 7B to 8B range are the workhorses of local AI. They answer quickly, run on hardware people already own, and are good enough for summarizing, drafting, classification, simple coding help and retrieval-based assistants. The question "will it run on my card?" has a precise answer, though, and it depends on three things: the precision you load the weights at, the context length you need, and how many conversations you serve at once. This guide gives you the arithmetic for the two most common models in the class, Llama 3.1 8B and Qwen2.5 7B, and maps it onto 8, 12, 16 and 24 GB GPUs.

The two reference models

The Llama 3 paper lists the 8B model with 32 layers, a model dimension of 4,096, a feed-forward dimension of 14,336, 32 attention heads and 8 key/value heads. Rebuilding the parameter count from those figures (attention, SwiGLU feed-forward, and untied embedding and output matrices with the 128,000-token vocabulary) yields about 8.03B parameters. The head dimension is 4,096 / 32 = 128.

The Qwen2.5 technical report lists the 7B model with 28 layers and 28 query heads sharing 4 KV heads, a 128K-token context length and untied embeddings. The llama.cpp llama-bench documentation shows its parameter count as 7.62B. It also shows the Q4_K Medium file at 4.36 GiB, which is roughly 4.9 bits per weight. Qwen's head dimension is 128 (hidden size divided by heads).

Weights: what fits at each precision

The approximate bit widths come from the llama.cpp example and block formats; they differ somewhat between models and between file types. Our quantization guide explains why a "4-bit" file is closer to 4.9 bits.

KV cache for the small models

Using 2 × layers × KV heads × head dim × bytes: Llama 3.1 8B in BF16 costs 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token. Qwen2.5 7B costs 2 × 28 × 4 × 128 × 2 = 57,344 bytes (56 KiB) per token, less than half, thanks to its 4 KV heads. At a context of 8,192 tokens that is 1 GiB for Llama and 0.44 GiB for Qwen; at 32,768 tokens, 4 GiB and 1.75 GiB; at 131,072 tokens, 16 GiB and 7 GiB. The extended derivation is in KV cache memory explained.

Which card fits which configuration

Add weights, cache and roughly 1 to 2 GB for the CUDA context, compute buffers and (on a desktop) the display. These are example plans, not guarantees:

• 8 GB card: Only the 4-bit file fits comfortably. Llama 3.1 8B at 4.9 GB plus a 1 GiB cache for 8K tokens plus about 1.5 GB overhead is about 7.5 GB, right at the limit. Qwen2.5 7B at 4.7 GB with the same 8K context (0.44 GiB) leaves more room. Expect to keep contexts at 8K tokens or below. 8-bit (about 8.5 GB) does not fit.
• 12 GB card: 8-bit fits: 8.5 GB plus 1 GiB cache plus overhead is about 11 GB for Llama, so 8K contexts work but 32K does not. Or use 4-bit with a much longer context: 4.9 + 4.3 (32K tokens) + 1.5 is about 10.7 GB.
• 16 GB card: BF16 does not fit (16.1 GB of weights alone for Llama; Qwen's 15.2 GB leaves almost nothing). 8-bit works well: 8.5 + 4.3 (32K tokens) + 1.5 is about 14.3 GB. A 4-bit file with a 128K context is also feasible for Qwen: 4.7 + 7.5 + 1.5 is about 13.7 GB.
• 24 GB card: Full BF16 fits. Llama 3.1 8B at 16.1 GB plus a 32K-token cache (4.3 GB) plus 1.5 GB is about 21.9 GB. A full 128K-token Llama session (17.2 GB of cache) does not fit in BF16; use an 8-bit cache or a quantized model. NVIDIA lists the GeForce RTX 4090 with 24 GB of GDDR6X.

The 24 GB tier also opens the 27B class. A nominal 27B model at about 4.9 bits per weight is roughly 16.5 GB, leaving room for a moderate context; try the Gemma 2 27B calculator for your own settings.

Speed ceilings for a single user

Token generation for one user is bounded by memory bandwidth divided by the bytes read per token, which for a dense model is essentially the weight file. With roughly 1 TB/s of bandwidth (about what an RTX 4090 provides; check the spec of your card):

• BF16 Llama 3.1 8B: 16.1 GB per token gives a ceiling near 62 tokens per second.
• 8-bit: 8.5 GB gives about 118 tokens per second.
• 4-bit: 4.9 GB gives about 204 tokens per second.

Real engines reach a fraction of these, and speeds drop with long contexts because the cache also has to be read. The point is the shape: halving the bytes roughly doubles the ceiling. A card with half the bandwidth gets half the ceiling regardless of how many CUDA cores it has. We derive the formula and its limits in inference speed explained. For a data-center comparison, an H100 SXM at 3.35 TB/s (NVIDIA's figure) would have a BF16 ceiling around 208 tokens per second on the same model.

Serving several users from a small model

A 24 GB card can do more than chat with one person. With vLLM and a utilization setting of 0.9, usable memory is about 21.6 GB. Subtract 16.1 GB of Llama weights and about 5.5 GB remains for cache and activations; assuming 1 GB for activations leaves 4.5 GB, or 4.5 × 10⁹ / 131,072 ≈ 34,000 tokens of cache. At 4,000 tokens per session that is about eight simultaneous conversations. With Qwen2.5 7B (57,344 bytes per token) the same budget holds about 78,000 tokens, roughly 19 sessions. Switching the cache to FP8, which vLLM supports, doubles both. These are example figures; the server prints its real capacity on startup. Batching also raises total throughput substantially while each user's speed falls, a trade-off covered in vLLM versus llama.cpp.

What else limits a small-model setup

• System RAM and loading time. Loading a 16 GB BF16 file requires fast storage and enough RAM to stage it; slow disks make start-up take minutes.
• CPU and PCIe. For a single GPU running a fully offloaded model, the CPU is rarely the limit during generation, but tokenization, sampling and prompt handling can show up at very high token rates; see CPU and GPU bottlenecks.
• Power supply. An RTX 4090 is listed at 450 W total graphics power with 850 W recommended system power on NVIDIA's page; the PSU calculator helps check a build.
• Quality ceiling. A 7B or 8B model has limited factual recall and reasoning compared with 70B. Test on your tasks before committing; if you outgrow it, see running 70B models locally.
• Portable devices. For battery-powered edge hardware, memory is shared and power is limited; see Jetson Orin power and battery runtime.

How to verify a fit in ten minutes

Paper arithmetic gets you to a short list; a quick test settles it. First download the file and note its size on disk, because for a quantized model the file size is almost exactly the weight memory. Second, start the engine with the context length you actually need, not the model's maximum. In llama.cpp the context is set with --ctx-size and the number of layers placed on the GPU with -ngl; the startup log reports how much memory each buffer takes, including the KV cache. Third, watch GPU memory with nvidia-smi while you send a long prompt, since some buffers are only allocated when a prompt is processed. If memory climbs to the card's limit, reduce the context, quantize the cache, or move to the next smaller weight format.

It is also worth knowing the failure modes. An out-of-memory error at load time means the weights plus the cache reservation exceed the card. An error only after a long prompt means activation buffers or cache growth was the problem. Slowdowns with no error often mean part of the model quietly spilled into system memory, which is dramatically slower than VRAM; if your tokens per second is a fraction of the ceiling you calculated, check that every layer is actually on the GPU. Keep a note of the configuration that works, including the model file, quantization, context length and engine version, so you can reproduce it after an update.

Which to choose

If you have a 24 GB card, run BF16 or 8-bit and spend the spare memory on context or concurrent users. With 16 GB, use 8-bit. With 12 GB, use 8-bit at short context or 4-bit at longer context. With 8 GB, use 4-bit and keep context modest. Between the two models, Qwen2.5 7B needs less cache per token while Llama 3.1 8B has a larger vocabulary and more parameters; benchmark both on your prompts, since quality differences depend on language and task. For a quick sanity check of your own settings, use the Qwen 2.5 7B VRAM calculator.

Sources

• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3
• Qwen2.5 Technical Report, Table 1
• llama.cpp llama-bench README (Qwen2 7B Q4_K Medium example)
• NVIDIA GeForce RTX 4090 specifications
• NVIDIA H100 data sheet
• vLLM documentation: Quantized KV cache

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.