LLM GPU Memory Calculator: Weights, KV Cache and GPU Count

Related Engineering Guides

• A100 vs H100 vs H200 vs B200: What Actually Changes for LLM Work — Compare NVIDIA A100, H100, H200 and B200 on memory, bandwidth, power and NVLink, and learn which number decides each LLM workload.
• CPU and GPU Bottlenecks Explained: How to Tell Which One Limits You — A practical guide to bottlenecks: why one number cannot label a pairing, how to read utilization, and how it differs between games and LLM inference.
• Deploying Llama 3.1 405B: GPU Counts for BF16, FP8 and 4-bit — Worked memory and bandwidth math for serving Llama 3.1 405B on H100 and H200 nodes, including Meta's own 16-GPU BF16 setup and FP8 on a single node.
• Fine-Tuning Memory Requirements: Full Training vs LoRA vs QLoRA — Why full fine-tuning needs about 16 bytes per parameter, how LoRA and QLoRA cut that, with worked numbers for Llama 3.1 8B and 70B.
• FP8, INT8 and 4-bit Quantization: What You Save and What You Risk — How quantization changes LLM memory and speed, how FP8, INT8, GPTQ, AWQ and GGUF formats differ, and how to size a model after quantizing it.
• Hardware for DeepSeek-V3 and R1: Why a 671B MoE Model Needs So Much Memory — What 671B total and 37B active parameters mean for GPU memory, bandwidth and interconnect, with FP8 sizing for 8x H100, 8x H200 and larger clusters.
• How to Estimate GPU Memory for an LLM: Weights, KV Cache and Overhead — A step-by-step method to estimate the VRAM an LLM needs for inference, with worked numbers for Llama 3.1 8B, 70B and 405B from the published architecture.
• KV Cache Memory Explained: Why Long Chats Eat Your VRAM — What the KV cache stores, the exact formula for its size, and worked numbers for Llama 3.1, Qwen2.5 and DeepSeek-V3 with GQA and MLA.
• NVLink vs PCIe: Reading the Bandwidth Numbers Correctly — Compare NVLink and PCIe bandwidth per generation, avoid the bidirectional-versus-per-direction trap, and learn when interconnect speed changes LLM results.
• Running a 70B Model at Home: Dual GPUs, CPU Offload and Realistic Speeds — How much memory a quantized 70B model needs, which home-lab layouts fit, and why layer splitting and RAM offload set your tokens-per-second.
• Tensor vs Pipeline Parallelism: Choosing How to Split a Model Across GPUs — How tensor and pipeline parallelism divide a model, what each costs in communication and idle time, and how to choose for inference or training.
• vLLM, llama.cpp and TensorRT-LLM: Choosing an Inference Server — Compare vLLM, llama.cpp and TensorRT-LLM on hardware support, batching, quantization and multi-GPU scaling to pick the right LLM server for your setup.
• What a 128K Context Window Really Costs in Memory and Compute — How long contexts inflate KV cache, prefill compute and decode time, with worked numbers for Llama 3.1 8B and 70B and the techniques that tame them.
• Which GPU Do You Need for a 7B or 8B Language Model? — VRAM needs for Llama 3.1 8B and Qwen2.5 7B at BF16, 8-bit and 4-bit on 8, 12, 16 and 24 GB cards, with context length and speed ceilings worked out.
• Why Your LLM Is Slow: Predicting Tokens per Second from Memory Bandwidth — A practical model of LLM inference speed: bandwidth-bound decoding, compute-bound prefill, batching, and speculative decoding, with worked H100 and RTX 4090 estimates.