Home › Guides › GPU memory for LLMs

FP8, INT8 and 4-bit Quantization: What You Save and What You Risk

Updated October 21, 2026 · By the Xingyu Zhitong editors

Quantization means storing a model's numbers with fewer bits. It is the most effective single lever for making a large model fit on a smaller GPU, and also the one most often misunderstood. People quote "4-bit means a quarter of the memory" and then find their file is bigger than expected, or that the model answers noticeably worse on their task. This guide explains the formats you will actually meet, gives sizing arithmetic that includes the overhead, and describes where Meta and the open-source tooling say the risks lie.

The basic idea and the basic arithmetic

A trained model is a set of weights, normally held in 16-bit floating point (BF16 or FP16), two bytes each. Quantization maps those weights to a smaller set of values: 8-bit integers or 8-bit floats (one byte), or 4-bit integers (half a byte). Memory for weights then scales directly:

weight bytes = parameters × bits per weight / 8

For a 70.55B-parameter model (the Llama 3.1 70B count from its published architecture) that gives 141 GB at 16 bits, 70.6 GB at 8 bits and 35.3 GB at an ideal 4 bits. Because the arithmetic is linear, quantizing is the fastest way to move a model across a hardware threshold: 141 GB needs two 80 GB cards, 70.6 GB barely squeezes onto one but leaves no room for cache, and 35 GB fits on a single 48 GB card with space to spare. Our memory estimation guide shows how to budget the rest.

Why 4-bit is not exactly 4 bits

Most low-bit formats quantize weights in small blocks and store a scale (and sometimes a minimum or zero point) for each block. Those extra values count toward file size. The llama.cpp llama-bench documentation includes a real example: a Qwen2 7B model with 7.62B parameters in "Q4_K - Medium" format occupies 4.36 GiB. That is 4.36 × 1.0737 × 10⁹ × 8 / 7.62 × 10⁹, or about 4.9 bits per weight. So when you plan for a "4-bit" GGUF, use about 4.5 to 5 bits per weight, not 4, and budget memory accordingly. Some tensors, such as the embedding table or output head, are often kept at higher precision as well.

The same example shows how to turn the number into a plan. A 70.55B model at 4.9 bits per weight is about 43 GB, not 35 GB. That detail matters when you are deciding between a 48 GB card and a 40 GB one.

The formats you will see

FP8 (E4M3 and E5M2)

FP8 stores each weight as an 8-bit floating-point number. NVIDIA's Hopper generation, including the H100 and H200, has Tensor Cores with native FP8 support, so FP8 saves both memory and compute time, not just memory. Meta used FP8 for Llama 3.1 405B inference. In the Llama 3 paper they quantize most parameters and activations in the feed-forward layers, which account for roughly 50 percent of inference compute time, but not the self-attention parameters; they skip the first and last transformer layers; they cap dynamic scaling factors at 1200; and they use row-wise rather than tensor-wise scaling. They also report that standard benchmarks suggested FP8 matched BF16 even without these safeguards, while without the cap the model occasionally produced corrupted responses. The lesson is that benchmark parity does not prove safety on rare inputs such as dates, so test your own workload.

In vLLM, FP8 is available through the quantization documentation and the LLM Compressor library, with both weight-only and weight-and-activation (W8A8) modes. Hardware support varies by GPU generation, so check the compatibility table in the vLLM docs before assuming an older card benefits.

INT8

8-bit integers are the older, widely supported option. The LLM.int8() paper (Dettmers et al., 2022) showed that a 175B-parameter checkpoint could be converted to Int8 and used immediately, halving the memory needed for inference without performance degradation, by handling a few "outlier" feature dimensions in 16-bit. INT8 W8A8 is also available through LLM Compressor in vLLM.

4-bit: GPTQ, AWQ, bitsandbytes NF4, GGUF

Four-bit formats are where memory savings are largest and quality risk grows. GPTQ (Frantar et al.) and AWQ (Lin et al.) are post-training methods that choose quantized weights to minimize error on calibration data; AWQ protects the small fraction of weights that matter most to activations. Both are supported in vLLM. bitsandbytes provides NF4, the 4-bit NormalFloat type introduced with QLoRA, mainly used for fine-tuning and convenient loading. GGUF is the file format of llama.cpp; its names (Q8_0, Q5_K_M, Q4_K_M and so on) encode the block scheme and bit width, and the llama-quantize tool produces them from a full-precision file, optionally guided by an importance matrix.

The 4.9-bit row applies the measured ratio from the llama-bench example to a different model, so treat it as an estimate; real file sizes vary by model.

Weights are not the only thing you can quantize

There are three separate targets, and mixing them up causes confusion:

• Weights (what most people mean). Saves static memory and bandwidth.
• Activations (the W8A8 style). Required to use low-precision Tensor Core math, so this is where speed gains beyond bandwidth come from.
• KV cache. A separate setting that shrinks per-session memory; vLLM exposes FP8 KV cache with optional calibration. For long contexts this can matter more than weight quantization. See the KV cache guide.

llama.cpp has equivalent cache settings, -ctk and -ctv (cache type for keys and values), in its server documentation.

Speed: why quantization helps token generation

Generating tokens for a single user is limited mostly by how fast the GPU can read the weights from memory, not by arithmetic. Each token requires reading essentially all the weights once, so halving the bytes per weight roughly doubles the ceiling on tokens per second, as long as your kernels can use the compact format efficiently. That is a bandwidth argument, and it holds even on cards without special low-precision math. The ceiling and its caveats are worked through in our speed guide. Quantization helps less when you batch many requests, because then compute becomes the limit, and the cost of dequantizing can offset the gain unless the kernel is designed for it.

A worked example: fitting Llama 3.1 405B

The Llama 3 paper states that the BF16 405B model does not fit on one machine with eight H100 GPUs, so Meta ran it across 16 GPUs on two machines using pipeline parallelism, and then used FP8 to reduce the requirement. The arithmetic shows why. Using the 405.8B parameters from the published architecture, BF16 needs 812 GB against 640 GB on eight 80 GB cards. FP8 needs about 406 GB, leaving roughly 234 GB across the node for cache, activations and overhead. With a 10 percent reserve (64 GB), about 170 GB remains for cache, which at 516,096 bytes per token in BF16 is about 330,000 tokens, or roughly two and a half full 128K-token sessions. Details are in the Llama 405B hardware guide, and you can try your own numbers in the 405B VRAM calculator.

Choosing a format: a decision checklist

1. Are you serving many users on Hopper-class GPUs? Start with FP8 weights and activations, with an FP8 KV cache if contexts are long, and evaluate on your own prompts.
2. Do you need to fit a model on one consumer or workstation card? Use a 4-bit weight format (AWQ or GPTQ in vLLM, or a GGUF 4- or 5-bit file in llama.cpp) and plan for roughly 4.5 to 5 bits per weight.
3. Is quality critical (code, math, structured output)? Prefer 8-bit or the highest-bit 4-bit variant you can afford, and compare against the 16-bit model on a fixed test set before deploying.
4. Are you fine-tuning? Quantize the frozen base only, as QLoRA does; see the fine-tuning memory guide.

Risks and how to test for them

Quantization error does not spread evenly. Benchmarks that average over many questions can hide failures on rare token patterns, long contexts, non-English text or precise arithmetic. The Llama 3 authors found exactly this with FP8. A sensible test is a fixed set of 100 to 200 prompts from your real workload, run on both the full-precision and quantized model, comparing task accuracy and checking for malformed output. Smaller models usually lose more from aggressive quantization than large ones, so the same 4-bit recipe that is harmless on 70B may visibly hurt on 7B. If a quantized model behaves oddly, try the next higher bit width before debugging anything else.

Finally, check licensing and provenance: quantized community uploads are separate files from the original release. Prefer quantizing from the official weights yourself, or use a publisher whose method and calibration data are documented. You can also size a mid-range case with the Gemma 2 27B calculator to see how 8-bit and 4-bit move it between card classes.

Sources

• The Llama 3 Herd of Models, Section 6.2 FP8 quantization (Dubey et al., 2024)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022)
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• vLLM documentation: Quantization
• llama.cpp llama-quantize README
• llama.cpp llama-bench README

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.