Home › Guides › Hardware, power and cost

A100 vs H100 vs H200 vs B200: What Actually Changes for LLM Work

Updated October 21, 2026 · By the Xingyu Zhitong editors

Four NVIDIA data-center GPU generations are still in circulation at the same time: Ampere (A100), Hopper (H100 and H200) and Blackwell (B200). Spec sheets list dozens of numbers for each, and most of them do not matter for a given job. For large language model work, three numbers do almost all of the explaining: memory capacity, memory bandwidth, and the interconnect between GPUs. This article lines those up from NVIDIA's own product pages and shows how to turn them into practical sizing decisions.

The numbers from NVIDIA's product pages

The table below uses figures published on NVIDIA's A100, H100, H200 and DGX B200 pages at the time of writing. NVIDIA labels some of these as preliminary or configurable, so treat them as the vendor's headline values, not measured results. Blackwell per-GPU values are derived by dividing the published eight-GPU system totals by eight, which is why they are marked as derived.

Two things stand out. First, between A100 and H100, memory capacity did not change at 80 GB, but bandwidth rose from roughly 2 TB/s to 3.35 TB/s. Second, H200 is essentially an H100 compute die with a much bigger and faster memory system: 141 GB and 4.8 TB/s, which NVIDIA describes as about 1.4 times the H100's bandwidth. If your workload is memory-bound, H200 is a different kind of upgrade than B200, which changes the compute generation as well.

Why bandwidth predicts single-stream decode speed

During token-by-token generation at batch size 1, each new token requires reading essentially every model weight from GPU memory once. The arithmetic is tiny compared with the data movement, so the time per token is close to model size divided by memory bandwidth. This gives an upper bound, never a prediction, because real kernels do not reach peak bandwidth and the KV cache adds more reads. The guide on tokens per second covers the rest of the picture.

Example, using the table above and a hypothetical 70-billion-parameter model quantized to FP8 (one byte per parameter, so about 70 GB of weights):

• H100 SXM: 3,350 GB/s divided by 70 GB is about 48 tokens per second upper bound.
• H200 SXM: 4,800 GB/s divided by 70 GB is about 69 tokens per second upper bound.
• A100 80GB SXM: Ampere has no native FP8 tensor math, so you would use INT8 or a weight-only format. The same 70 GB gives 2,039 divided by 70, about 29 tokens per second.

These ratios, not the absolute values, are what to carry away: H200 over H100 is about 1.43 times, matching the bandwidth ratio, while H100 over A100 is about 1.64 times. Notice also that a 70 GB weight file technically fits an 80 GB card but leaves only about 10 GB for the KV cache, CUDA context and activations, which limits batch size and context length severely. That is the practical reason capacity matters as much as bandwidth.

Capacity decides how many GPUs you need

Capacity sets the minimum GPU count. A 70B model in FP16 is about 140 GB of weights (70 billion times two bytes). On 80 GB cards that means at least two GPUs before any cache; on H200's 141 GB a single card is technically just short of holding weights plus any KV cache, so FP8 or two cards is the realistic choice. Our 70B FP16 VRAM calculator shows the breakdown including overhead, and the memory estimation guide explains each term.

Fewer GPUs per replica has second-order benefits. Splitting a model across GPUs with tensor parallelism adds communication on every layer (see tensor vs pipeline parallelism), so a model that fits on one H200 can run with no inter-GPU traffic at all, where the same model on 80 GB cards needs two or four-way splitting. The extra capacity can therefore pay back in efficiency, not just in headroom.

Where compute generation matters

Bandwidth explains decode. Prompt processing (prefill) and training are different: they process many tokens per weight read, so they are limited by tensor-core throughput. Here the generation matters more than the memory. Hopper introduced the Transformer Engine with FP8 support; Blackwell extends the supported low-precision formats. NVIDIA's HGX B200 page quotes 72 petaflops of FP8/FP6 for an eight-GPU board with sparsity, and states that the dense figure is half the sparse one, so about 36 petaflops dense. Marketing comparisons often quote the sparse number, which most real workloads cannot exploit, so check which one a chart uses before comparing GPUs.

A rough rule: if your workload is interactive chat with small batches, rank GPUs by bandwidth. If it is large-batch serving, fine-tuning or pre-training, rank by usable low-precision throughput and interconnect. For batch serving, a larger KV-cache budget also lifts throughput because it allows more concurrent sequences; the guide on KV cache memory shows how fast it grows.

Interconnect: SXM versus PCIe cards

NVIDIA quotes 900 GB/s of NVLink per H100 SXM or H200 SXM GPU, 600 GB/s for A100, and 1.8 TB/s for Blackwell. Those are totals counting both directions. The PCIe Gen5 x16 interface listed beside them is 128 GB/s, also a bidirectional total. So within one HGX node, GPU-to-GPU bandwidth is roughly seven times higher than going over PCIe on Hopper. The PCIe-form-factor cards (H100 NVL, H200 NVL, A100 PCIe) use bridges that connect only two or four GPUs, which is fine for splitting one model across a pair but not a substitute for a full NVSwitch fabric. The article on NVLink versus PCIe explains when this changes the outcome.

Power and what it does to the budget

Power is the number buyers underweight. An H100 SXM can draw up to 700 W, so an eight-GPU board is 5.6 kW before CPUs, NICs, fans and storage; NVIDIA's DGX B200 page lists about 14.3 kW maximum for the full system. That has three consequences: rack power and cooling must be planned before purchase (see air versus liquid cooling), the electricity bill is not negligible (see PUE and electricity cost), and cost per token depends on efficiency, not just peak speed. You can plug your own tariff and utilization into the HGX 8-GPU electricity calculator.

A worked comparison: one model, three cards

Suppose you want to serve a hypothetical 70B model in FP8 with a modest 8,000-token context for a handful of concurrent users. Weights are about 70 GB. Per-token KV cache depends on the architecture; with grouped-query attention it is a few hundred kilobytes per token, so 8,000 tokens per user is on the order of a few gigabytes, and ten concurrent users is tens of gigabytes. The exact figure comes from the formula in the KV cache guide; the point here is the shape of the result.

• On a single 80 GB card the weights consume most of memory, so you can serve perhaps one or two sequences. Two cards with tensor parallelism give room for batching but add communication.
• On a single 141 GB H200 the same weights leave roughly 60 GB or more for cache and overhead, enough for a useful batch with no inter-GPU traffic, and bandwidth per card is higher.
• On Blackwell with about 180 GB per GPU (derived from the DGX B200 totals), the headroom is larger still, and the same model could in principle share the card with other work.

This is why the question "which GPU is fastest" is usually the wrong one. The better question is "which configuration holds my model plus cache with the fewest GPUs, and what does a GPU-hour of that configuration cost me". The second half of that sentence is covered in the guide on renting versus buying GPUs.

A decision checklist

1. Write down the model size and precision. Compute weights in GB, then add KV cache for your batch and context length.
2. Find the smallest GPU count whose total memory covers that with at least 10 to 20 percent spare. Prefer a count that avoids cross-GPU splitting if one bigger-memory card covers it.
3. If latency per user matters, compare bandwidth. If aggregate throughput matters, compare low-precision compute and memory for batching.
4. If you will split across GPUs, check the interconnect: SXM with NVLink for tensor parallelism across four or eight GPUs, PCIe or bridges for two-way splits or pipeline parallelism.
5. Check the supported precision: FP8 needs Hopper or newer, so A100 deployments rely on FP16, BF16 or INT8. The quantization guide explains the trade-offs.
6. Add power, cooling and software support to the budget. A cheaper GPU that needs more units may cost more in total.

Caveats about spec sheets

Headline numbers are peak values under ideal conditions. NVIDIA marks some figures, including the H200 page, as preliminary and subject to change, and configurable power limits mean two systems with the same GPU may behave differently. Memory bandwidth in practice reaches a fraction of peak, and the fraction varies by kernel. If a decision rides on a margin of ten or twenty percent, benchmark your own model on the serving stack you plan to use; framework choice (see inference server comparison) can move results more than the GPU generation gap between adjacent products.

Sources

• NVIDIA A100 product page
• NVIDIA H100 product page
• NVIDIA H200 product page
• NVIDIA HGX platform page
• NVIDIA DGX B200 page
• NVIDIA NVLink page

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.