Home › Guides › Speed and scaling

vLLM, llama.cpp and TensorRT-LLM: Choosing an Inference Server

Updated October 21, 2026 · By the Xingyu Zhitong editors

Once you know how much GPU memory a model needs, the next decision is the software that will actually run it. Three open-source projects dominate self-hosted language-model serving: vLLM, llama.cpp and NVIDIA's TensorRT-LLM. They overlap in what they can do, but they were built for different situations, and the wrong choice can cost you more throughput than a GPU generation upgrade would gain. This article compares them on the dimensions that matter and gives a way to decide. Features change quickly, so treat this as a map and check each project's current documentation before committing.

vLLM: throughput for many concurrent requests

vLLM started as a research project built around PagedAttention, described in a 2023 paper by Kwon et al. The idea: the KV cache, which grows with every token of every active sequence, is stored in fixed-size blocks that need not be contiguous, much like virtual memory pages. This avoids the waste that comes from reserving a maximum-length contiguous buffer per request, so more sequences fit in the same memory, and larger batches mean more tokens per second per GPU. The sizing logic behind that is in KV cache memory explained.

Key characteristics, according to the project's documentation:

• Continuous batching: new requests join the running batch between decoding steps instead of waiting for a whole batch to finish.
• OpenAI-compatible API server: existing client code can point at it with a changed base URL.
• Multi-GPU: supports tensor parallelism and pipeline parallelism through launch settings; see the parallelism article for what those mean for performance.
• Quantization: supports a range of formats, including FP8 on supporting hardware; the supported list differs by version and hardware, so check the quantization page of the docs.
• Hardware: built primarily for NVIDIA GPUs, with support for other accelerators that varies by release.

Best for: serving many users or batch jobs on one or more data-center GPUs, where aggregate throughput and cost per token matter. Weak spots: it expects a GPU with enough memory for the whole model (or the whole shard); it is not designed for running a model that spills into CPU RAM; and Python-heavy startup and dependency management can be awkward on small machines.

llama.cpp: runs almost anywhere

llama.cpp is a C/C++ implementation written to run models on ordinary hardware. Its model format, GGUF, stores weights together with metadata, and the project provides a family of quantization types (the names such as Q4_K_M and Q8_0 encode bit width and scheme) that shrink models substantially. The repository documents backends for CPUs, NVIDIA GPUs through CUDA, Apple silicon through Metal, and cross-vendor GPUs through Vulkan, among others.

Its signature feature is partial offload: you tell it how many layers to place on the GPU (the --n-gpu-layers option), and the rest run on the CPU from system memory. That lets a laptop or a single 24 GB card run a model larger than its VRAM, at a cost in speed, since the CPU layers read weights at system-memory bandwidth and the GPU layers at GPU bandwidth. It includes a built-in HTTP server (llama-server) with an OpenAI-style API, supports multiple concurrent slots, and can split a model across several GPUs.

Best for: single-user or small-team use, desktops, laptops, Macs, mixed hardware, and any scenario where memory is the constraint and some slowdown is acceptable. Weak spots: at high concurrency on a data-center GPU it generally delivers lower aggregate throughput than engines built for batching (benchmark to see by how much on your hardware), and the quantized formats trade some quality for size; see our quantization guide.

TensorRT-LLM: maximum performance on NVIDIA hardware

TensorRT-LLM is NVIDIA's open-source library for optimized LLM inference on NVIDIA GPUs. Its documentation describes in-flight batching, paged KV caching, multi-GPU and multi-node parallelism, and low-precision support including FP8 on Hopper and newer architectures. A distinguishing trait is the build step: the model is compiled into an engine optimized for a particular GPU architecture and configuration, and changing certain settings such as the parallel layout means rebuilding.

Best for: production deployments on NVIDIA data-center GPUs where the last increment of performance justifies the engineering. It tends to track new NVIDIA hardware features first. Weak spots: NVIDIA-only; more setup effort; engines are tied to the GPU type and library version, which makes operations more rigid. Check the supported-model matrix in the docs before assuming your model works.

Side-by-side summary

How to choose

1. What hardware do you have? A Mac or a PC without a data-center GPU points to llama.cpp. NVIDIA data-center GPUs open all three options.
2. Does the model fit entirely in GPU memory? If not and you cannot add GPUs, llama.cpp's partial offload is the practical choice. Check capacity with the memory estimation guide or the 70B VRAM calculator first.
3. How many simultaneous requests? One to three users: llama.cpp is simple and fast enough. Dozens or more: vLLM or TensorRT-LLM.
4. How much engineering time do you have? Start with the easiest tool that meets your target, and move up only after a measurement shows a gap.
5. Do you need to change models often? vLLM and llama.cpp load models directly; TensorRT-LLM requires rebuilding engines, which suits stable, long-lived deployments.

A worked example: where the difference comes from

Consider a 7B model in 16-bit precision, weights near 14 GB (7 billion parameters × 2 bytes), on a single GPU with 3.35 TB/s of memory bandwidth, like the H100 SXM that NVIDIA lists. At batch size 1 each decoding step must read about 14 GB, so the bandwidth bound is roughly 3,350 ÷ 14 ≈ 239 tokens per second for one user. Going to batch size 32, the weights are still read once per step and shared by all 32 sequences, so the aggregate rate can rise by a large factor, limited by compute and KV cache reads rather than weights. That gap between single-stream and batched throughput is why batching-focused engines show their advantage under load and little at one user. An engine that cannot keep the batch full, or that wastes cache memory so fewer sequences fit, leaves that gain on the table. More background on this is in tokens per second, and the single-GPU sizing is in running 7B models on one GPU.

For a user with a single RTX 4090 (24 GB per NVIDIA's page) running a 4-bit quantized 7B or 13B model for personal use, llama.cpp with a GGUF file is typically the simplest route and gives interactive speeds. The same machine running vLLM would also work for models in supported formats but offers little advantage unless you serve several users at once.

Benchmarking honestly

Published comparisons between engines age quickly because each project ships performance work frequently. If the decision matters, run your own test:

• Use your real model, quantization and prompt/output lengths. Short prompts and long prompts stress different parts of the system.
• Measure three things separately: time to first token (prefill latency), tokens per second per user (decode speed), and aggregate tokens per second at your target concurrency.
• Test at several concurrency levels, such as 1, 8 and 32, because rankings often change with load.
• Pin versions of the engine, driver and CUDA libraries and record them, so a result can be reproduced.
• Check output quality, not just speed. Different quantization schemes and kernels can change answers slightly.
• Record GPU power and temperature; a thermally throttled run misleads (see cooling).

Operational points that decide long-term satisfaction

Beyond raw speed, consider how the server behaves in production. Does it expose metrics you can scrape? Does it handle request cancellation, so abandoned requests free their cache? How does it behave when the cache is full: queue, preempt, or fail? How quickly does it add support for new model architectures when they are released? vLLM and llama.cpp both add new architectures quickly in practice because of large communities, while TensorRT-LLM support follows NVIDIA's release schedule, so check the support matrix for the exact model you want. Also check licensing for both the software and the model weights; the engines are open source, but model licenses vary and are separate.

Finally, remember that the engine sits on top of hardware limits. Neither software choice can fix a model that does not fit memory, or an interconnect too slow for the chosen parallelism (see NVLink versus PCIe). Pick the hardware by memory and bandwidth first, as covered in the GPU comparison, then pick the software to fit.

Sources

• vLLM documentation
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
• llama.cpp repository and documentation
• TensorRT-LLM documentation
• NVIDIA H100 product page
• NVIDIA GeForce RTX 4090 product page

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.