Home › Guides

Xingyu Zhitong Guides

In-depth guides to sizing AI and GPU hardware: how much memory a model needs, what limits inference speed, how GPUs compare, and what a deployment costs to run. The worked examples use published model configurations and vendor datasheets, linked in each guide.

Hardware, power and cost

• Renting Cloud GPUs vs Buying Your Own: A Break-Even MethodA step-by-step method for comparing GPU rental with ownership, with every cost term listed and a worked example using clearly labeled assumptions.
• PUE and the Real Electricity Cost of Running GPU ServersWhat PUE means, how to turn GPU wattage into a monthly power bill, and a worked example for an 8-GPU server using example tariffs.
• Cooling GPU Servers: Air vs Liquid, With the Heat Math Worked OutHow to convert GPU wattage into airflow or coolant flow, when air cooling runs out of room, and what liquid cooling changes for racks and operations.
• A100 vs H100 vs H200 vs B200: What Actually Changes for LLM WorkCompare NVIDIA A100, H100, H200 and B200 on memory, bandwidth, power and NVLink, and learn which number decides each LLM workload.
• Jetson AGX Orin Battery Runtime: Power Modes, Losses and a Worked EstimateEstimate how long a Jetson AGX Orin runs a vision workload on battery, covering power modes, system overhead, converter losses and battery derating.
• Sizing a Power Supply for an RTX 4090 Build: ATX 3.x ExplainedHow to size a PSU for an RTX 4090 and a high-end CPU, what ATX 3.0 and 3.1 add, and how to avoid melted-connector and transient-trip problems.

Speed and scaling

• CPU and GPU Bottlenecks Explained: How to Tell Which One Limits YouA practical guide to bottlenecks: why one number cannot label a pairing, how to read utilization, and how it differs between games and LLM inference.
• vLLM, llama.cpp and TensorRT-LLM: Choosing an Inference ServerCompare vLLM, llama.cpp and TensorRT-LLM on hardware support, batching, quantization and multi-GPU scaling to pick the right LLM server for your setup.
• Why Your LLM Is Slow: Predicting Tokens per Second from Memory BandwidthA practical model of LLM inference speed: bandwidth-bound decoding, compute-bound prefill, batching, and speculative decoding, with worked H100 and RTX 4090 estimates.
• NVLink vs PCIe: Reading the Bandwidth Numbers CorrectlyCompare NVLink and PCIe bandwidth per generation, avoid the bidirectional-versus-per-direction trap, and learn when interconnect speed changes LLM results.
• Tensor vs Pipeline Parallelism: Choosing How to Split a Model Across GPUsHow tensor and pipeline parallelism divide a model, what each costs in communication and idle time, and how to choose for inference or training.

GPU memory for LLMs

• Hardware for DeepSeek-V3 and R1: Why a 671B MoE Model Needs So Much MemoryWhat 671B total and 37B active parameters mean for GPU memory, bandwidth and interconnect, with FP8 sizing for 8x H100, 8x H200 and larger clusters.
• Fine-Tuning Memory Requirements: Full Training vs LoRA vs QLoRAWhy full fine-tuning needs about 16 bytes per parameter, how LoRA and QLoRA cut that, with worked numbers for Llama 3.1 8B and 70B.
• KV Cache Memory Explained: Why Long Chats Eat Your VRAMWhat the KV cache stores, the exact formula for its size, and worked numbers for Llama 3.1, Qwen2.5 and DeepSeek-V3 with GQA and MLA.
• Deploying Llama 3.1 405B: GPU Counts for BF16, FP8 and 4-bitWorked memory and bandwidth math for serving Llama 3.1 405B on H100 and H200 nodes, including Meta's own 16-GPU BF16 setup and FP8 on a single node.
• How to Estimate GPU Memory for an LLM: Weights, KV Cache and OverheadA step-by-step method to estimate the VRAM an LLM needs for inference, with worked numbers for Llama 3.1 8B, 70B and 405B from the published architecture.
• FP8, INT8 and 4-bit Quantization: What You Save and What You RiskHow quantization changes LLM memory and speed, how FP8, INT8, GPTQ, AWQ and GGUF formats differ, and how to size a model after quantizing it.
• What a 128K Context Window Really Costs in Memory and ComputeHow long contexts inflate KV cache, prefill compute and decode time, with worked numbers for Llama 3.1 8B and 70B and the techniques that tame them.
• Running a 70B Model at Home: Dual GPUs, CPU Offload and Realistic SpeedsHow much memory a quantized 70B model needs, which home-lab layouts fit, and why layer splitting and RAM offload set your tokens-per-second.
• Which GPU Do You Need for a 7B or 8B Language Model?VRAM needs for Llama 3.1 8B and Qwen2.5 7B at BF16, 8-bit and 4-bit on 8, 12, 16 and 24 GB cards, with context length and speed ceilings worked out.

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy.