Home › Guides › GPU memory for LLMs

Hardware for DeepSeek-V3 and R1: Why a 671B MoE Model Needs So Much Memory

Updated October 21, 2026 · By the Xingyu Zhitong editors

DeepSeek-V3 and DeepSeek-R1 are often described with two numbers that seem to contradict each other: 671 billion parameters and 37 billion activated per token. Which one decides the hardware you need? The honest answer is both, but for different resources. Memory capacity follows the total, while the speed of each generated token follows the active part. Understanding that split is the key to sizing a deployment, and to understanding why reports about "running R1" range from a single workstation to hundreds of GPUs. This guide works through the architecture published by DeepSeek and turns it into hardware numbers.

What the official sources say

The DeepSeek-V3 technical report describes a Mixture-of-Experts (MoE) model with 671B total parameters, 37B activated for each token. The architecture details from its hyper-parameter section are:

• 61 transformer layers, hidden dimension 7,168.
• Multi-head latent attention (MLA) with 128 heads of dimension 128, a KV compression dimension of 512, and a decoupled rotary dimension of 64.
• All feed-forward layers except the first three are MoE layers. Each has 1 shared expert and 256 routed experts, with an expert intermediate dimension of 2,048. Eight routed experts are activated for each token.
• Multi-token prediction with depth 1.

The model's GitHub README lists DeepSeek-V3 and DeepSeek-V3-Base at 671B total, 37B activated and a 128K context length, and notes that the total size on Hugging Face is 685B, comprising 671B of main model weights plus 14B of Multi-Token Prediction module weights. The DeepSeek-R1 README lists R1 and R1-Zero with the same 671B / 37B / 128K figures. It also lists six distilled dense models of 1.5B, 7B, 8B, 14B, 32B and 70B, based on Qwen2.5 and Llama 3, which are separate and much smaller models; "running R1" on a laptop almost always means one of those distills rather than the 671B model.

Memory follows the total, not the active parameters

An MoE router picks different experts for different tokens, and over a batch of requests nearly all experts get used. The server therefore has to keep every expert resident in memory. Weight memory is:

weight bytes = total parameters × bytes per parameter

The model was trained with an FP8 mixed-precision framework (the report says this was the first validation of FP8 training at this scale), and the repository's reference inference configuration lists "dtype": "fp8", so FP8 is the natural reference precision rather than a lossy compromise. Treat BF16 as an expansion you would only do for specific reasons.

Now compare with common GPU nodes, using the published memory sizes of 80 GB for the H100 and 141 GB for the H200 from NVIDIA's data sheets:

• 8 × H100 (640 GB): FP8 weights alone (671 GB) exceed the node. It does not fit without going below 8 bits.
• 8 × H200 (1,128 GB): 685 GB of FP8 weights leaves 443 GB. Subtracting a 10 percent reserve (113 GB) leaves roughly 330 GB for cache and activations.
• 16 × H100 (1,280 GB, two nodes): fits with about 595 GB spare before the reserve, but the weights now cross a network link between nodes.

The H200 point deserves emphasis: its 141 GB per GPU, against 80 GB, is what lets a 671B FP8 model fit in a single eight-GPU node. See the H100, A100, H200 and B200 comparison for the cards side by side.

The KV cache is unusually small

Thanks to MLA, the per-token cache is tiny for a model this large. With 576 cached elements per layer (512 + 64) across 61 layers, a BF16 cache costs 576 × 61 × 2 = 70,272 bytes (about 68.6 KiB) per token. Continuing the 8 × H200 example with 330 GB available: 330 × 10⁹ / 70,272 is about 4.7 million tokens, or around 36 full 128K-token sessions. The same 330 GB on a Llama 3.1 405B-style cache (504 KiB per token) would hold only about 640,000 tokens. So for DeepSeek, the weights are the entire capacity problem; the cache is a secondary concern. See KV cache memory explained for why. (Engines may lay out the latent cache differently, so confirm with your engine's reported token capacity.)

Speed follows the active parameters, with caveats

Each decoded token activates 37B parameters. At FP8 that is about 37 GB read per token if only the needed experts are fetched, compared with 671 GB for the whole model. At a single sequence, token generation is limited by memory bandwidth, so a simple ceiling is bandwidth / bytes read per token. For eight H200s at 4.8 TB/s each (NVIDIA's published figure), the combined 38.4 TB/s divided by 37 GB gives a theoretical ceiling near 1,000 tokens per second for one stream. Real systems come in far lower, because experts live on different GPUs, so each layer requires all-to-all communication, because kernels do not reach peak bandwidth, and because attention and routing add their own costs. Use the ceiling only for comparing designs, not for promising performance.

With a large batch, the picture changes. Many tokens in the batch route to many different experts, so the engine ends up reading most of the model each step, and the useful measure becomes total throughput across all requests, which is why DeepSeek's own serving architecture is so large.

How DeepSeek serves it at scale

The V3 report describes a production deployment that separates prefill and decoding. The minimum prefill unit is 4 nodes with 32 GPUs, using 4-way tensor parallelism with sequence parallelism for attention, 8-way data parallelism, and 32-way expert parallelism for the MoE layers. The minimum decoding unit is 40 nodes with 320 GPUs: 4-way tensor parallelism plus 80-way data parallelism for attention, and 320-way expert parallelism where each GPU hosts only one expert (64 GPUs host redundant and shared experts). Tokens travel between nodes over InfiniBand and then within a node over NVLink.

Two lessons follow for smaller deployments. First, MoE inference is communication-heavy, so the interconnect matters more than for dense models of similar active size; compare NVLink with PCIe before choosing consumer cards. Second, the parallelism strategy is different from the dense-model default; the trade-offs between splitting inside layers and across layers are in tensor versus pipeline parallelism. The report's own numbers are for a very large, latency-optimized service, not a recommendation for a lab.

Why the dense layers and shared expert matter

Not every parameter is an equal citizen. The first three layers use ordinary dense feed-forward blocks (the repository's reference configuration lists an intermediate dimension of 18,432 for them), and every MoE layer has one shared expert that every token passes through. These always-on parts, together with attention and the embedding table, are a large share of the 37B activated parameters. In a multi-GPU layout they are typically replicated or split by tensor parallelism, while the 256 routed experts per layer are spread across devices. That asymmetry is why hot spots appear: if the router sends many tokens to a few experts, the GPUs holding them become the bottleneck while others idle. DeepSeek-V3 addresses this with an auxiliary-loss-free load-balancing strategy during training and redundant expert copies at serving time, but if you build your own deployment, watch per-GPU utilization, not just the aggregate tokens per second.

Running it on less: what trades off

Several routes exist for smaller budgets, each with costs:

• Lower-bit quantization. At an ideal 4 bits the main model is about 336 GB, which could in principle sit on 8 × H100 (640 GB). Real 4-bit files are larger because of block scales (the llama.cpp example in our quantization guide averages about 4.9 bits per weight, giving roughly 411 GB), and quality must be tested. Aggressive community quantizations exist; this site has not verified their accuracy.
• CPU and system-memory offload. Because only 37B parameters are active per token, engines such as llama.cpp can keep rarely used experts in system RAM. Capacity is solved, but speed then tracks RAM bandwidth. A dual-channel DDR5-5600 desktop moves at most 2 × 8 bytes × 5,600 million transfers per second, about 90 GB/s, so reading even 37 GB per token at 4.9 bits (roughly 23 GB) would cap throughput at only a few tokens per second.
• Distilled models. If your goal is reasoning ability on modest hardware, the 7B to 70B distilled R1 models fit on single GPUs. They are different models with different quality; see running 70B models locally.

Power and operating cost

An eight-GPU H100 or H200 node draws a few kilowatts under load; the NVIDIA data sheet lists a maximum thermal design power of up to 700 W per SXM GPU, so the GPUs alone approach 5.6 kW before CPUs, networking and cooling. If you plan to run such a node continuously, estimate electricity with the HGX 8-GPU electricity cost calculator and compare renting with buying in cloud GPU rent versus buy. To size the memory side for your own settings, use the DeepSeek-V3 VRAM calculator or the DeepSeek-R1 version, and remember that reasoning models like R1 generate long outputs, which lengthens sessions and raises cache demand.

Summary of the sizing rules

1. Plan memory from 671B (or 685B with the MTP module) times bytes per parameter.
2. Reserve roughly 10 percent plus activation space; the cache itself is small thanks to MLA.
3. Expect speed to depend on active parameters, expert placement and interconnect, and measure it.
4. Confirm any quantized variant on your own tasks before relying on it.

Sources

• DeepSeek-V3 Technical Report
• DeepSeek-V3 GitHub repository and README
• DeepSeek-R1 GitHub repository and README
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V2 Technical Report (MLA)
• NVIDIA H100 data sheet
• NVIDIA H200 data sheet

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.