Home › Guides › GPU memory for LLMs

Running a 70B Model at Home: Dual GPUs, CPU Offload and Realistic Speeds

Updated October 21, 2026 · By the Xingyu Zhitong editors

A 70-billion-parameter model is the sweet spot many enthusiasts aim for: clearly stronger than the 7B to 8B class, yet small enough that a single desktop can, with compromises, run it. The compromises are what people underestimate. A 70B model quantized to four bits still needs over 40 GB, which is more than any single consumer GPU offers, so you must split it across cards, spill into system memory, or use a machine with a large pool of unified memory. Each route has a different speed ceiling, and you can compute that ceiling before spending money. This guide shows how.

How big is a 70B model, really?

Using the architecture in Table 3 of the Llama 3 paper (80 layers, model dimension 8,192, feed-forward dimension 28,672, 64 attention heads, 8 key/value heads), the Llama 3.1 70B parameter count works out to about 70.55B. Weight size then depends on precision:

The 4.9-bit figure is derived from the llama.cpp llama-bench documentation, where a 7.62B-parameter model in Q4_K Medium takes 4.36 GiB; the 8.5-bit figure reflects the 8-bit block format storing one 16-bit scale per 32 weights. Both are approximations that vary by model. The full discussion is in our quantization guide. For home use, the practical target is the roughly 43 GB four-bit file. A Llama 3.3 70B release is in the same size class; check its config.json for the layer and head counts before you rely on these numbers.

Then add the KV cache: 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KiB) per token in BF16. At 8,192 tokens that is 2.5 GiB; at 32,768 tokens, 10 GiB. Storing the cache at 8 bits halves those figures. See the KV cache guide.

Layout 1: two 24 GB GPUs

NVIDIA lists the GeForce RTX 4090 with 24 GB of GDDR6X, so two cards give 48 GB. The 43 GB four-bit file plus a 2.5 GiB cache at 8K context is about 45.7 GB, leaving roughly 2.3 GB for two CUDA contexts, compute buffers and the desktop. That is extremely tight, and in practice you will use a shorter context (4K tokens needs 1.25 GiB), an 8-bit cache, or a slightly smaller quantization. Anything larger than Q4 does not fit.

In llama.cpp, the server documentation lists the relevant options: -ngl (--n-gpu-layers) to put layers on the GPUs, --split-mode (none, layer, row, tensor), --tensor-split to set the proportion per card, --ctx-size for context, and -ctk / -ctv to choose the cache types. A typical layer-split run offloads all layers (a value higher than the layer count, such as 99, is common) and divides them evenly between the cards. Quantizing the V cache generally requires flash attention to be enabled; confirm in the README for your build.

What speed to expect. With layer splitting, each token passes through the first card's layers and then the second's, one after the other. Only one card is busy at a time, so the single-stream speed ceiling is not the sum of the two cards' bandwidth. If each card reads half the weights (about 21.5 GB) at roughly 1 TB/s (the RTX 4090's memory bandwidth is about 1 TB/s; check your card's specification), the time per token is about 21.5 ms per card, or 43 ms in total, a ceiling near 23 tokens per second. Real results are lower. The benefit of two cards is capacity, not speed. Only small activation tensors cross between the cards in layer mode, so a PCIe link is adequate; compare to the heavy traffic of tensor parallelism in NVLink versus PCIe.

Layout 2: one GPU plus system RAM

If you have a single 24 GB card and 64 GB or more of RAM, llama.cpp can keep some layers on the GPU and the rest on the CPU. Capacity is solved; speed is not. The time per token is the sum of reading the GPU-resident part at GPU bandwidth and the CPU-resident part at RAM bandwidth. Here is an example with assumptions stated:

• Total weights: 43 GB. Suppose 20 GB fit on the GPU (leaving room for cache and buffers) and 23 GB stay in RAM.
• GPU part: 20 GB at about 1,000 GB/s takes 0.020 s.
• CPU part: dual-channel DDR5-5600 transfers at most 2 channels × 8 bytes × 5,600 million transfers per second, about 90 GB/s. Reading 23 GB takes about 0.256 s.
• Total: about 0.276 s per token, or roughly 3.6 tokens per second as a ceiling.

The slow part dominates by a factor of more than ten, so adding more GPU layers helps in proportion to the share of the model they move off the CPU, and a faster GPU alone helps very little. That is also why people see 1 to 3 tokens per second on CPU-only 70B runs. It is usable for batch jobs and overnight tasks, tolerable for patient chat, and frustrating for coding assistance.

Layout 3: large unified-memory machines

Some desktop and laptop systems share one memory pool between CPU and GPU, so a 64 GB or larger configuration can hold the 43 GB model with ample room for cache. Capacity is then easy, and speed again follows the machine's memory bandwidth divided by 43 GB. Because bandwidth differs enormously between models and generations, look up the vendor's published figure for the exact configuration and apply the same division, instead of trusting a generic claim. Also confirm how much of the pool the operating system allows the GPU to use; defaults are often a fraction of the total.

Prompt processing is a separate bottleneck

The tokens-per-second ceilings above describe generation. Reading a long prompt (prefill) behaves differently: the model processes many tokens at once, so it is limited by arithmetic throughput rather than memory bandwidth. The cost is about two floating-point operations per parameter per token, so a 70B model needs roughly 140 billion operations for every prompt token. That is where offloading to a CPU hurts most, because CPUs deliver far less matrix throughput than GPUs. A 20,000-token document that a dual-GPU setup digests in a modest wait can take many minutes on a CPU-heavy layout. If your use involves long documents, measure prompt speed separately with llama-bench, whose README shows prompt-processing (pp) and token-generation (tg) tests as separate columns, instead of relying on a single tokens-per-second figure.

It is also worth deciding up front how much context you really need. Many local sessions never exceed 4,000 to 8,000 tokens, and keeping the context window at that size saves several gigabytes of cache on a 70B model, which can be the difference between fitting in 48 GB and not.

Power, PSU and noise

NVIDIA's RTX 4090 page lists a total graphics power of 450 W and a required system power of 850 W for one card. Two cards mean planning for around 900 W of GPU power alone, so a single 850 W supply is not enough and you need a high-wattage unit with enough native connectors, plus airflow for two thick cards. Our PSU sizing guide for the RTX 4090 and ATX 3 covers connectors and transient spikes, and the RTX 4090 PSU calculator estimates a single-card build. Card thickness also matters physically: two 3- or 4-slot cards often cannot fit side by side on a standard motherboard without riser cables.

Choosing among the options

If those speeds are not acceptable, consider the alternatives: a smaller dense model such as the 7B to 8B class on a single GPU, which runs several times faster on the same card, or renting a data-center GPU by the hour; see renting versus buying. To check memory for your own context length and precision, use the Llama 3.3 70B calculator or the Llama 3.1 70B calculator.

A short checklist before you buy

1. Decide the quantization you will accept (Q4 is the realistic floor for quality on most tasks) and compute the file size using about 4.9 bits per weight.
2. Add cache for your real context length, using 320 KiB per token for BF16 cache.
3. Compute the speed ceiling as bandwidth divided by bytes read per token, per layout, and halve it as a planning figure.
4. Check power supply, slot spacing and cooling.
5. Test the model on your own prompts in a rented instance first, so you know the quality justifies the hardware.

Sources

• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3
• NVIDIA GeForce RTX 4090 specifications
• llama.cpp server README (GPU layers, split mode, cache types)
• llama.cpp llama-bench README
• llama.cpp llama-quantize README
• GQA: Grouped-Query Attention (Ainslie et al., 2023)

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.