Home › Guides › GPU memory for LLMs

Fine-Tuning Memory Requirements: Full Training vs LoRA vs QLoRA

Updated October 21, 2026 · By the Xingyu Zhitong editors

Running a model and training it are very different memory problems. A GPU that serves a 7B model comfortably can run out of memory immediately when you try to fine-tune the same model, because training must hold extra copies of the numbers: gradients, optimizer statistics and the intermediate values needed for backpropagation. Parameter-efficient methods, LoRA and its quantized cousin QLoRA, exist largely to shrink that burden. This guide builds the memory budget from first principles, then compares three approaches with numbers for Llama 3.1 8B and 70B.

Where training memory goes

Four things sit in GPU memory during training:

1. Model weights. Same as inference.
2. Gradients. One value per trainable parameter, produced by backpropagation.
3. Optimizer states. Adam keeps two extra values per trainable parameter (a running mean and a running variance of the gradient), and mixed-precision setups also keep a full-precision master copy of the weights.
4. Activations. Intermediate outputs from the forward pass that the backward pass needs. These scale with batch size, sequence length and depth, not with parameter count alone.

The ZeRO paper (Rajbhandari et al., 2020) gives the standard accounting for mixed-precision training with Adam: with Ψ parameters, the model needs 2Ψ bytes for FP16 weights, 2Ψ bytes for FP16 gradients, and KΨ bytes for optimizer states, where K = 12 (a 4-byte FP32 master copy, a 4-byte momentum and a 4-byte variance). That totals 16Ψ bytes before activations.

Full fine-tuning: 16 bytes per parameter

Using that rule, here is the arithmetic for Llama 3.1 8B (about 8.03B parameters, from the architecture in Table 3 of the Llama 3 paper): 8.03 × 10⁹ × 16 bytes = 128.5 GB. That is already more than an 80 GB H100 before any activations. For 70B (about 70.55B parameters): 70.55 × 10⁹ × 16 = 1,129 GB, about 14 H100s' worth of memory, which is why 70B full fine-tuning is done with sharded optimizers across many GPUs (ZeRO or similar techniques that partition states across devices). A rough practical statement: full fine-tuning of an 8B model needs a multi-GPU node or memory-saving tricks such as 8-bit optimizers, while 70B needs a cluster.

LoRA: freeze the base, train small adapters

Low-Rank Adaptation (Hu et al., 2021) freezes the pretrained weights and injects small trainable low-rank matrices into the layers. For a weight matrix of shape d_in × d_out, LoRA with rank r adds two matrices totalling r × (d_in + d_out) parameters. The paper reports that, compared with fine-tuning GPT-3 175B using Adam, LoRA can reduce the trainable parameter count by 10,000 times and the GPU memory requirement by 3 times, with quality on par or better on the models they tested.

Count the adapters for Llama 3.1 8B with rank r = 16 applied to all seven linear projections in each layer (query, key, value, output, gate, up, down). With model dimension 4,096, KV projection width 8 × 128 = 1,024 and feed-forward width 14,336:

• Query: 4,096 + 4,096 = 8,192
• Key: 4,096 + 1,024 = 5,120; Value: 5,120
• Output: 8,192
• Gate, up and down: each 4,096 + 14,336 = 18,432, so 55,296 for the three

The sum per layer is 81,920; multiplied by r = 16 that is 1,310,720; multiplied by 32 layers it is 41.9 million trainable parameters, about 0.5 percent of the model. Training state for these adapters at 16 bytes each (4-byte weight, 4-byte gradient, 8 bytes of Adam statistics, assuming FP32 adapters) is 0.67 GB. The memory budget for LoRA on 8B in BF16 is therefore:

• Frozen base weights, BF16: 16.1 GB
• Adapter weights, gradients and optimizer states: about 0.7 GB
• Activations: depends on batch and sequence length, often several GB

That fits on a 24 GB card with a modest batch, whereas full fine-tuning did not fit on 80 GB. The same calculation for 70B at rank 16 gives about 207 million adapter parameters (3.3 GB of training state), but the frozen BF16 base alone is 141 GB, so LoRA on 70B still requires multiple large GPUs or quantization of the base.

QLoRA: quantize the frozen base to 4 bits

QLoRA (Dettmers et al., 2023) backpropagates gradients through a frozen, 4-bit quantized base model into LoRA adapters. Per the paper's abstract, it reduces memory enough to fine-tune a 65B model on a single 48 GB GPU while preserving full 16-bit fine-tuning performance, using three ideas: a 4-bit NormalFloat (NF4) data type, double quantization of the quantization constants, and paged optimizers that manage memory spikes. The authors fine-tuned more than 1,000 models in their analysis.

Apply it to our two models. Llama 3.1 8B base at about 4.5 to 5 bits per weight (4-bit block formats carry extra scale data; see our quantization guide) is roughly 4.5 to 5 GB. Add 0.7 GB of adapter state and activations, and a 12 GB card becomes realistic for short sequences. Llama 3.1 70B at the same bit width is roughly 40 to 43 GB of weights, which is why a 48 GB card is the practical entry point, with a small batch and gradient checkpointing, consistent with the paper's 65B result. The cost of QLoRA is speed: weights must be dequantized on the fly, so training is slower than plain LoRA on the same hardware.

The LoRA and QLoRA rows add the base weights and the adapter training state, using the adapter counts computed above. Activations come on top and can easily add 2 to 20 GB depending on batch size and sequence length, so these are floors.

Activations: the part the formulas do not give you

Activation memory grows with batch size, sequence length, and the number of layers. It is why a configuration that fits at 512 tokens can fail at 4,096. Three controls matter most:

• Micro-batch size. Lower it and use gradient accumulation to keep the effective batch size. Memory falls nearly in proportion, at no cost in final quality, only in speed.
• Gradient checkpointing. Discard most activations in the forward pass and recompute them in the backward pass. It saves a large share of activation memory in exchange for extra compute, commonly noted as around a third more time.
• Sequence length. Truncate or pack data sensibly. Doubling the length doubles per-layer activations, and standard attention adds quadratic score matrices unless a memory-efficient kernel such as FlashAttention is used; see long-context memory cost.

There is no honest closed formula that applies to every framework, so measure: run one step with your batch and sequence length, read the peak allocated memory, and scale your hyperparameters from there.

Choosing rank and target layers

Rank and target modules change memory only slightly, because adapters are small compared with the base. Using the 8B counts above, rank 8 would halve the 41.9 million adapter parameters to about 21 million, saving roughly 0.3 GB of training state, while rank 64 would quadruple them to about 168 million and cost about 2.7 GB. Applying adapters only to the query and value projections shrinks the count much more, since the feed-forward projections hold most of the dimensions, but the LoRA and QLoRA papers and later practice found that adapting more layers generally helps quality. The practical advice is to start at a modest rank on all linear layers, check validation loss against a held-out set, and raise the rank only if the model underfits. Spending your memory budget on a larger micro-batch or a longer sequence is usually a better use than on a higher rank.

Finally, remember that evaluation and generation during training also consume memory. Running a validation pass with a long generation can briefly need more than a training step, so leave a margin of a gigabyte or two rather than sizing to the last megabyte, and save checkpoints of the adapters often; they are tiny compared with a full model.

Choosing a method

1. Need a style, format or domain adaptation on a few thousand examples? Start with LoRA or QLoRA. These are the typical use cases.
2. Memory is tight? Use QLoRA, accept the slower step time, and use gradient checkpointing.
3. Need to add substantial new knowledge or change behaviour deeply? Adapters may not suffice; full fine-tuning or continued pretraining requires much more hardware and data.
4. Deployment. LoRA adapters can be kept separate and loaded on demand, or merged into the base weights; merged models then follow normal inference sizing, covered in GPU memory estimation.

Renting versus owning for training

Fine-tuning is a bursty workload, which usually favours renting. A QLoRA run on an 8B model can finish in hours on a single rented GPU. Before buying hardware for occasional runs, compare costs in cloud GPU rent versus buy and test numbers in the cloud versus on-prem ROI calculator. For inference-side sizing of the resulting model, use the Llama 3.1 70B or Qwen 2.5 7B calculators, and for the card options see the GPU comparison.

Sources

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Rajbhandari et al., 2020)
• The Llama 3 Herd of Models (Dubey et al., 2024), Table 3
• FlashAttention: Fast and Memory-Efficient Exact Attention (Dao et al., 2022)

Figures are estimates based on published model configurations and vendor specifications. Real memory use, speed and power draw depend on the software stack, drivers, settings and workload, so test on your own hardware before you buy. Spotted a mistake? Email huhuwen09@gmail.com with the page address and we will check and correct it.