Home › Guides › Speed and scaling

Tensor vs Pipeline Parallelism: Choosing How to Split a Model Across GPUs

Updated October 21, 2026 · By the Xingyu Zhitong editors

When a model does not fit on one GPU, or when one GPU is too slow, you have to split the work. The two most common ways to split a single model are tensor parallelism and pipeline parallelism. They are not interchangeable: they cut the model along different axes, stress different parts of the hardware, and fail in different ways. Picking the wrong one can leave expensive GPUs waiting on each other. This article explains how each works, what each costs, and how to choose, with a worked communication estimate.

Tensor parallelism: split inside each layer

In tensor parallelism, the large matrices inside each transformer layer are partitioned across GPUs. The Megatron-LM paper (Shoeybi et al., 2019) describes the standard approach: the first linear layer of the feed-forward block is split by columns and the second by rows, and attention heads are distributed among GPUs, so each GPU computes a slice of the layer independently. The partial results must be combined, which takes an all-reduce collective. In that design each transformer layer needs two all-reduce operations in the forward pass (one after attention, one after the feed-forward block) and two more in the backward pass during training.

Consequences:

• Every GPU participates in every layer, so all GPUs work at the same time. There is no idle time from the splitting itself.
• Memory for weights is divided roughly by the number of GPUs, and so is the per-GPU share of the KV cache in inference when attention heads are split.
• Communication happens many times per token or per training step, on the critical path. Its cost depends directly on interconnect bandwidth and latency.
• The tensor-parallel degree is limited by structure, for example by the number of attention heads or key-value heads, which must divide evenly. Frameworks document these constraints.

Pipeline parallelism: split between layers

In pipeline parallelism, consecutive groups of layers are assigned to different GPUs ("stages"). A request or micro-batch flows through stage 1, then stage 2, and so on. The only data crossing between GPUs is the activation tensor at stage boundaries, which is much less communication than tensor parallelism's per-layer all-reduces. It can work over slower links, even across nodes connected by InfiniBand or Ethernet.

The cost is idle time, known as the pipeline bubble. With one big batch, only one stage works at a time while the rest wait. To reduce this, the batch is divided into micro-batches so that stage 1 can start micro-batch 2 while stage 2 handles micro-batch 1. The Megatron-LM scaling paper (Narayanan et al., 2021) gives the bubble fraction for the standard schedule as (p − 1) / m, where p is the number of pipeline stages and m the number of micro-batches. With 4 stages and 16 micro-batches the bubble is 3/16, about 19 percent of the time; with 4 stages and 4 micro-batches it is 3/4, a poor use of hardware. The same paper also describes interleaved schedules that reduce the bubble at the cost of more communication.

Consequences:

• Memory for weights is divided across stages, but activations and the micro-batch queue add complexity.
• Inter-GPU traffic is small, making it suitable for PCIe-connected GPUs or multiple nodes.
• At low request rates (for example one user chatting), the bubble cannot be hidden: a single token must pass through all stages in sequence, so latency per token is the sum of stage times plus transfers, not a fraction of them.
• Stages should have balanced work; uneven layers or an embedding-heavy first stage cause stalls.

A worked communication estimate

Take a model shaped like Llama 70B for illustration: 80 layers, hidden size 8,192, 16-bit activations. These values are published for the Llama 3 70B architecture; verify against the model card for your version. For tensor parallelism across 8 GPUs, there are 2 all-reduces per layer, so 160 per forward pass.

Decode (one token per step). Each all-reduce carries one token's hidden vector: 8,192 × 2 bytes = 16 KB. That is a tiny message, so time is dominated by latency, not bandwidth: 160 collectives per generated token, each costing perhaps tens of microseconds on NVLink (an assumption to measure on your system) and noticeably more over PCIe. At 20 µs each, communication is 3.2 ms per token, which is a meaningful fraction of a roughly 20 to 30 ms token time; at 100 µs each over a slower path it is 16 ms, a large fraction. Decode with tensor parallelism is therefore latency-sensitive, not bandwidth-sensitive.

Prefill (4,096 prompt tokens). Each all-reduce carries 4,096 × 16 KB = 64 MB. Over 160 all-reduces that is 10.5 GB of tensor data. A ring all-reduce moves about 2 × (N − 1) / N times the tensor size per GPU, which for N = 8 is 1.75, so about 18 GB per GPU. NVIDIA quotes 900 GB/s of NVLink bandwidth per H100 SXM GPU as a bidirectional total, so roughly 450 GB/s each way: 18 GB / 450 GB/s ≈ 41 ms. Over PCIe Gen5 at about 64 GB/s per direction the same traffic takes about 290 ms. The compute for the prompt itself is on the order of 2 × 70 × 10⁹ × 4,096 ≈ 5.7 × 10¹⁴ floating-point operations, which eight GPUs at a realistic few hundred teraflops each finish in the order of a few hundred milliseconds. So on NVLink the communication is a modest tax; over PCIe it rivals the compute. This is the core reason tensor parallelism is paired with fast interconnects; see NVLink versus PCIe for the link numbers.

For pipeline parallelism with 8 stages, each boundary sends one activation per micro-batch (16 KB per token) once: 7 transfers per token rather than 160 collectives. That traffic is trivial even on PCIe or Ethernet. The cost shows up elsewhere: unless many requests are in flight, the other stages are idle.

Combining them

Large deployments use both, plus data parallelism (replicating the whole setup to serve more requests). The standard guidance from the Megatron-LM scaling work is to use tensor parallelism within a node where NVLink is fast, and pipeline parallelism across nodes where bandwidth is lower. For example, a model needing 16 GPUs could use tensor-parallel size 8 within each node and pipeline-parallel size 2 across two nodes. Inference servers expose these as settings: vLLM documents tensor and pipeline parallel sizes, and TensorRT-LLM supports both as well. See our server comparison for how the engines differ.

Mixture-of-experts models add a third dimension, expert parallelism, where different experts live on different GPUs and tokens are routed with all-to-all communication. The hardware implications are discussed in DeepSeek V3 and R1 hardware.

What the serving engines expose

You rarely implement these schemes yourself. In vLLM the tensor-parallel and pipeline-parallel sizes are launch options, and the product of the two (times any data-parallel replicas) must match the GPUs you give it. TensorRT-LLM takes the same choices when you build an engine, so the parallel layout is fixed at build time and must be rebuilt if you change it. llama.cpp takes a different route: it can split a model's layers across several GPUs, which resembles a simple pipeline split by layer, and it can also place some layers on the CPU. Always read the current documentation for your version, because supported combinations, such as pipeline parallelism together with certain quantization formats, change between releases.

Two practical details save time. First, check that the model's number of attention heads and key-value heads divides by your tensor-parallel size; grouped-query attention models often have only eight key-value heads, which caps useful tensor parallelism near eight without replicating cache. Second, leave a memory margin on every GPU: each process holds its own CUDA context, communication buffers and activation workspace, so the usable memory per GPU is less than the card's nominal capacity. The KV cache guide shows how the remaining space turns into concurrent sequences.

Choosing: a practical decision guide

1. Does the model fit on one GPU with room for KV cache? If yes, do not split it. Run replicas instead (data parallelism) and avoid communication entirely. Check with the memory estimation guide.
2. Do you have NVLink or NVSwitch between the GPUs? If yes, tensor parallelism up to the node size is usually the first choice for latency-sensitive serving.
3. Only PCIe between GPUs? Prefer the smallest tensor-parallel degree that fits (often 2) or pipeline parallelism, and expect lower per-user speed. Consumer RTX 4090 cards have no NVLink, so this is the common case on workstations.
4. Crossing nodes? Use pipeline parallelism across nodes; keep tensor parallelism inside them.
5. Serving many concurrent users? Pipeline bubbles shrink as more requests are in flight; throughput-oriented serving suits pipeline more than a single-user chat does.
6. Training? Use enough micro-batches that (p − 1) / m stays small, and consider interleaved schedules.
7. Measure. Run a short benchmark of each configuration with your batch size and context length. Published rules of thumb fail at the edges.

Common mistakes

Choosing a tensor-parallel degree that does not divide the number of key-value heads, so the framework refuses or silently wastes memory. Using eight-way tensor parallelism over PCIe and wondering why throughput is worse than four-way. Forgetting that tensor parallelism does not reduce per-GPU overhead such as CUDA context and activation buffers, so memory per GPU is not exactly total divided by N. Using too few micro-batches in pipeline training. And splitting a model that would fit on a single higher-memory GPU: a single H200 with 141 GB (see the GPU comparison) can sometimes replace a two-GPU tensor-parallel pair on 80 GB cards.

Sources

• Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019)
• Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021)
• vLLM documentation (parallelism and scaling)
• TensorRT-LLM documentation
• NVIDIA H100 product page (NVLink bandwidth)

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.