Home › Guides › Speed and scaling

NVLink vs PCIe: Reading the Bandwidth Numbers Correctly

Updated October 21, 2026 · By the Xingyu Zhitong editors

Interconnect bandwidth is the least understood line on a GPU spec sheet. Numbers like 900 GB/s and 128 GB/s sit side by side, but they are not directly comparable to the memory bandwidth figures next to them, and they are quoted on a different basis than many benchmark tools report. This article sorts out what the numbers mean, how large the NVLink-to-PCIe gap really is, and which workloads care.

Three different bandwidths in one GPU server

A GPU sees three data paths, each orders of magnitude apart:

1. GPU memory (HBM or GDDR): thousands of GB/s. NVIDIA lists 3.35 TB/s for H100 SXM and 4.8 TB/s for H200. This is the path used to read model weights during decoding.
2. GPU-to-GPU links (NVLink): hundreds of GB/s. NVIDIA lists 900 GB/s per H100 or H200 SXM GPU, 600 GB/s for A100, and 1.8 TB/s GPU-to-GPU for Blackwell in its HGX B200 platform.
3. Host link (PCIe): tens of GB/s. NVIDIA lists PCIe Gen5 at 128 GB/s on H100 and H200 pages.

Each step down is roughly an order of magnitude slower, which is the reason where data lives matters more than raw compute for many AI jobs. The full comparison of generations is in A100, H100, H200 and B200 compared.

Bidirectional totals versus per-direction speed

Both NVIDIA's NVLink figures and the 128 GB/s PCIe Gen5 figure are totals counting both directions. A PCIe 5.0 link runs at 32 GT/s per lane; across 16 lanes that is 32 × 16 / 8 = 64 GB/s in each direction (raw rate, ignoring encoding), so 128 GB/s combined. PCIe 4.0 is half of that: 32 GB/s per direction, 64 GB/s combined. PCI-SIG announced PCIe 5.0 in 2019 at 32 GT/s, and the generations double the rate each time. Real transfers also lose a few percent to protocol overhead, so measured throughput on a Gen5 x16 slot is somewhat under 64 GB/s each way.

Collective operations such as all-reduce use both directions at once (data flows around a ring in one direction on each link), so the combined figure is the right one to reason about only when traffic is balanced. For a one-way bulk copy, such as loading weights from host memory, use the per-direction number. When someone says "NVLink is seven times faster than PCIe", the comparison is 900 versus 128 GB/s on Hopper-class parts, a ratio of 7.0, and the same ratio holds if both are halved. For the older pair, A100's 600 GB/s against PCIe Gen4's 64 GB/s is about 9.4 times.

Topology matters as much as speed

NVLink bandwidth in a full HGX server is available between any pair of GPUs because the GPUs connect through NVSwitch chips. NVIDIA's DGX B200 page lists 14.4 TB/s of aggregate NVLink bandwidth for its eight GPUs with two NVSwitch chips. That is the design that makes eight-way tensor parallelism practical.

Other products connect GPUs differently. The PCIe-form-factor A100 and H100 NVL and H200 NVL cards use bridges: NVIDIA states the A100 PCIe bridge links only two GPUs, and the H200 NVL supports 2- or 4-way bridges. Two bridged cards communicate fast with each other, but a pair of pairs would fall back to PCIe between the pairs. Consumer GPUs from the RTX 40 series, including the RTX 4090, do not offer NVLink at all, so any multi-GPU communication goes over PCIe, and may traverse the CPU if the platform does not support direct peer-to-peer transfers between those cards.

You can inspect your own machine with nvidia-smi topo -m, which prints a matrix showing whether each pair of GPUs is connected through NVLink, through a PCIe switch, or across CPU sockets. Treat that matrix as the ground truth before you pick a parallelism strategy.

When the gap matters

Tensor parallelism. Every layer performs all-reduce operations on the critical path, so bandwidth and latency determine speed. This is the workload that most strongly favors NVLink. A simple model: a ring all-reduce of S bytes across N GPUs moves 2 × (N − 1) / N × S bytes through each GPU's link. With S = 64 MB (a 4,096-token prefill at hidden size 8,192 in 16-bit) and N = 4, that is 96 MB per all-reduce. At 450 GB/s per direction it takes about 0.21 ms; at 64 GB/s about 1.5 ms. Multiply by two all-reduces per layer and 80 layers and the totals diverge by a factor of seven, as the link ratio says. Details are in the parallelism article.

Pipeline parallelism. Only activations cross stage boundaries, which is small enough that PCIe is usually fine. The constraint is idle time, not bandwidth.

Single-GPU inference. The interconnect barely matters once the weights are loaded. A 14 GB model loads from host memory in well under a second at tens of GB/s; the disk is usually the slower part.

Offloading to CPU memory. When a model does not fit in VRAM and layers or KV cache spill into system RAM, every token must pull that data across PCIe. If 20 GB per token had to cross a 64 GB/s one-way PCIe 5.0 link, the bound would be about 3 tokens per second at best, and practical systems perform worse. That is why partly offloaded models feel slow, a point revisited in running 70B models locally.

Training. Gradient synchronization in data parallelism moves model-sized tensors every step, so fast interconnects matter more for larger models, though overlap of communication with computation hides much of it. Multi-node training adds InfiniBand or Ethernet fabrics, which are a further step down in bandwidth per GPU; that is a topic of its own.

Mixture-of-experts. Expert parallelism uses all-to-all exchanges, sensitive to both bandwidth and latency; see MoE hardware requirements.

Latency, not just bandwidth

Bandwidth tells you how fast a large transfer finishes; latency tells you how long a tiny transfer takes. Token-by-token decoding sends small messages, so per-message latency dominates there. NVLink and NVSwitch are designed for low-latency GPU-to-GPU memory access, while a PCIe transfer between two GPUs may route through a switch or through the CPU's root complex, adding hops. Exact latencies depend on the platform and are best measured, not quoted. A useful habit is to benchmark two message sizes: a few kilobytes, which exposes latency, and hundreds of megabytes, which exposes bandwidth. A link can look fine on the second test and poor on the first, and decode-heavy chat serving feels the first.

Peer-to-peer support is a related issue on consumer hardware. Some driver and platform combinations disable direct GPU-to-GPU PCIe transfers, forcing copies through system memory, which halves effective bandwidth and adds latency. If multi-GPU scaling on a workstation looks worse than the link speeds suggest, check whether peer-to-peer access is enabled, and whether both cards sit on lanes attached to the same CPU.

Buying perspective

Two cheaper PCIe cards are not equivalent to one NVLink-connected pair when the model must be split. A common workstation question is whether two 24 GB cards are as good as one 48 GB card. For a model that needs 40 GB, the two-card setup holds the weights but pays communication on every layer over PCIe, and each card holds duplicates of some activations. The single 48 GB card avoids that. Conversely, if your goal is throughput across independent requests and the model fits on one 24 GB card, two cards deliver nearly double the throughput with no communication at all. Compare the options with the memory estimation guide before deciding.

Common misreadings of the spec sheet

• Comparing an NVLink total of 900 GB/s with a memory bandwidth of 3.35 TB/s and concluding the link is "almost as fast as memory". It is about a quarter, and a collective operation also has to wait for the slowest participant.
• Adding NVLink and PCIe numbers together. They are separate paths used for different transfers, and software normally chooses one per operation.
• Assuming the bridge on a PCIe card gives the full-node NVLink fabric. The bridge connects two or four GPUs only.
• Ignoring that PCIe lanes are shared. Four GPUs plus several NVMe drives on a desktop platform often means x8 or x4 links, not x16.

Checking your own system

1. Run nvidia-smi topo -m and note the link types between each pair of GPUs.
2. Check the PCIe generation and width for each GPU in the monitoring tool while the card is busy (cards lower the link state at idle). A x16 slot negotiated down to x8 or x4 halves or quarters the host bandwidth.
3. Use NVIDIA's NCCL test suite (the open-source nccl-tests programs) to measure real all-reduce bandwidth; expect numbers below the theoretical peak.
4. Benchmark your serving stack with 1, 2 and 4 GPUs. If scaling is far below linear, communication is the likely culprit.

The interconnect also affects power and cooling in subtle ways: NVSwitch systems are dense, high-power machines (NVIDIA lists about 14.3 kW maximum for DGX B200), so planning for them should include cooling and not only link speed.

Sources

• NVIDIA NVLink overview
• NVIDIA H100 product page
• NVIDIA H200 product page
• NVIDIA A100 product page
• NVIDIA DGX B200 page
• PCI-SIG
• NVIDIA nccl-tests

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.