Home › Guides › GPU memory for LLMs

Deploying Llama 3.1 405B: GPU Counts for BF16, FP8 and 4-bit

Updated October 21, 2026 · By the Xingyu Zhitong editors

Llama 3.1 405B is the largest openly released dense language model most teams will ever try to host. Unlike a mixture-of-experts model, every one of its 405 billion parameters participates in every token, so both the memory footprint and the per-token memory traffic are enormous. The practical questions are concrete: how many GPUs, of which kind, at which precision, and what throughput can you expect? This guide answers them from Meta's published architecture and inference notes, with all arithmetic shown so you can substitute your own constraints.

The model in numbers

Table 3 of the Llama 3 paper lists the 405B configuration: 126 layers, a model dimension of 16,384, a feed-forward dimension of 53,248, 128 attention heads and 8 key/value heads, with a 128K-token vocabulary-sized embedding table and a context window of up to 128K tokens. Rebuilding the parameter count from those figures (attention projections, SwiGLU feed-forward, and untied embedding and output matrices) gives about 405.8B parameters, matching the name. Each head has dimension 16,384 / 128 = 128.

Weight memory at common precisions (decimal GB):

The KV cache, from the formula 2 × layers × KV heads × head dim × bytes, is 2 × 126 × 8 × 128 × 2 = 516,096 bytes (504 KiB) per token in BF16. A single 128K-token (131,072) session therefore holds 63 GiB of cache by itself. The derivation is in the KV cache guide.

What Meta did

The Llama 3 paper is explicit about the baseline. In BF16 the 405B model does not fit in the memory of a single machine with eight NVIDIA H100 GPUs: 8 × 80 = 640 GB against 812 GB of weights. Meta therefore parallelized BF16 inference across 16 GPUs on two machines. Within each machine, the high NVLink bandwidth supports tensor parallelism; across machines, where bandwidth is lower and latency higher, they used pipeline parallelism. The paper notes that pipeline "bubbles", a major issue in training, are not a concern during inference because there is no backward pass requiring a pipeline flush. That is a useful rule: put tensor parallelism inside the NVLink domain and pipeline parallelism across the slower links. The reasoning is expanded in tensor versus pipeline parallelism.

Meta then applied FP8 quantization to reduce the requirement. As described in the paper, they quantize most parameters and activations in the feed-forward layers (roughly half of inference compute time), leave the attention parameters unquantized, skip the first and last transformer layers, and bound dynamic scaling factors to 1200 with row-wise scaling to avoid corrupted outputs on high-activation tokens. At 406 GB, the FP8 weights fit comfortably on one eight-GPU H100 node.

Configuration options compared

For each option, usable memory assumes a 10 percent reserve for activations and overhead (an assumption; measure on your stack). Cache capacity uses BF16 cache entries at 516,096 bytes per token.

Memory sizes come from NVIDIA: 80 GB for the H100 and 141 GB for the H200. Several conclusions are visible. FP8 on eight H100s is the cheapest comfortable single-node setup, but 330,000 cache tokens is only about two and a half 128K-token sessions at once, or around 40 sessions of 8K tokens. Doubling cache capacity by storing the cache in FP8 is possible in vLLM at some accuracy risk. The H200's extra capacity makes BF16 possible on one node, which matters if you want to avoid quantization error entirely. The 4-bit four-GPU option fits but leaves little cache and needs quality validation.

Tensor parallel sizes and the KV head limit

Tensor-parallel sizes must divide the attention head count (128), and the model has only 8 key/value heads. With a tensor-parallel degree of 8, each GPU holds exactly one KV head and 1/8 of every weight matrix, which is a clean split; this is why eight-GPU nodes are the natural unit for this model. Going beyond 8-way tensor parallelism forces the engine to replicate KV heads or shard them awkwardly, wasting cache memory and adding communication. When you need more than eight GPUs, adding pipeline stages (as Meta did) or running separate replicas is the better scaling direction. Check the vLLM parallelism documentation for the exact behaviour in your version.

Per-GPU weight memory with FP8 and TP=8 is 406 / 8 = 50.7 GB, leaving about 29 GB of each 80 GB card before the reserve. The per-GPU view is what you will see in monitoring, so compare it with the totals above.

A sizing walk-through for a team of users

Suppose a team expects at most 20 simultaneous sessions, each averaging 6,000 tokens of prompt plus reply, on the 8 × H100 FP8 setup. The cache demand is 20 × 6,000 = 120,000 tokens, well under the roughly 330,000 the node can hold, so memory is not the constraint. Now suppose two of those users submit -token documents. The cache need becomes 18 × 6,000 + 2 × = 308,000 tokens, close to the limit, and any further long request will trigger waiting or recomputation. These are example numbers, but the pattern is general: the average session length hides the effect of a few long ones, so size for your long-document users explicitly.

When you launch the server, read its startup log for the number of cache tokens it actually allocated, then compare it with the table. If it is much lower than predicted, the usual causes are a lower memory-utilization setting, activation buffers sized for a large maximum batch of tokens, or a cache stored at higher precision than you assumed. Adjust one setting at a time and re-read the log.

Speed: bandwidth sets the ceiling

Dense decoding reads all weights once per generated token for a single stream. A simple ceiling is aggregate memory bandwidth divided by weight bytes. NVIDIA lists 3.35 TB/s for the H100 SXM and 4.8 TB/s for the H200.

• 8 × H100, FP8: 8 × 3.35 = 26.8 TB/s; 26,800 / 406 ≈ 66 tokens per second for one stream, as a theoretical ceiling.
• 8 × H200, BF16: 8 × 4.8 = 38.4 TB/s; 38,400 / 812 ≈ 47 tokens per second.
• 16 × H100, BF16 with TP8 × PP2: a single stream passes through both pipeline stages in turn, so only one stage's eight GPUs read memory at any moment, giving about 26,800 / 812 ≈ 33 tokens per second, no faster than a single node. Pipeline parallelism adds capacity, not single-stream speed.

Real numbers will be lower because of communication, kernel efficiency and scheduling. The ceilings are useful for comparing configurations, not for promising performance. Throughput across many users is a different measure: batching reuses each weight read for many tokens, so aggregate tokens per second can be many times higher than the single-stream figure until you become compute-limited. See inference speed explained for the model, and vLLM versus llama.cpp for the software side.

Interconnect and node requirements

Tensor parallelism needs two all-reduce operations per layer in the forward pass, so intra-node bandwidth is critical; eight GPUs on an NVLink/NVSwitch baseboard are the expected platform. Consumer cards linked only by PCIe will bottleneck badly; compare the numbers in NVLink versus PCIe. If you span two nodes with pipeline parallelism, traffic between stages is only the activation tensor at each stage boundary, far smaller than tensor-parallel traffic, so a good InfiniBand or fast Ethernet link is adequate, as Meta's design reflects.

Power, cooling and cost

NVIDIA lists a maximum TDP of up to 700 W for H100 SXM and H200 SXM GPUs. Eight of them are up to 5.6 kW before the host, fans and networking. Plan the facility accordingly, and see air versus liquid cooling. For annual electricity cost, try the HGX 8-GPU electricity calculator, and compare with renting using the cloud versus on-prem ROI calculator and our rent-versus-buy guide. For a hands-on memory check, use the Llama 3.1 405B VRAM calculator.

Is 405B the right choice?

Before buying eight data-center GPUs, test whether a smaller model meets the need. The 70B model in the same family needs a fraction of the hardware (see running 70B locally), and mixture-of-experts alternatives trade memory for per-token compute differently (see DeepSeek-V3 and R1). Run your real prompts through a hosted endpoint of the 405B and 70B models first. If the quality difference on your task is small, the 70B saves most of the capital and power bill. If it is large, the tables above give you a defensible starting configuration, to be refined with measurements on your own serving stack.

Sources

• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3 and Section 6 Inference
• Meta llama-models repository
• NVIDIA H100 data sheet
• NVIDIA H200 data sheet
• vLLM documentation: Parallelism and scaling
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.