LLM Fine-Tuning VRAM Calculator

An out-of-memory error can arrive on the first backward pass even when a language model loads successfully. Fine-tuning adapts a pretrained model using examples of a task, writing style or domain. The graphics processing unit (GPU) needs enough video memory, or VRAM, for the model and the work needed to learn from those examples. The saved model's file size therefore gives only part of the memory requirement.

During training, a forward pass produces predictions and intermediate values called activations. The backward pass uses those values to calculate gradients, which describe how trainable weights should change. An optimizer then updates the weights. AdamW maintains two running statistics, called moments, for each trainable parameter, and some mixed-precision arrangements also retain a higher-precision master copy of the trainable weights. These allocations can outweigh the model's working weights.

The training method determines how much of the model needs that additional training state.

Freezing the base model removes its parameter gradients and optimizer states, but training still passes information through it to learn the adapters. Activations remain part of the budget. Adapter rank controls the inner width of each trainable matrix pair. Raising the rank or adapting more projections increases the trainable parameter count. Extending adapters from attention to the feed-forward network also adds weights and training state. Lowering rank saves memory, while the effect on task quality needs evaluation.

The text being processed matters too. Sequence length counts tokens, the units into which text is split, and the micro-batch is the number of sequences processed together on one GPU. Long-document training can need much more temporary memory than short-response training on the same model. Longer sequences and larger micro-batches increase activations and the prediction scores, or logits, produced across the vocabulary. Conventional attention also stores relationships between token pairs, giving part of its memory demand a quadratic dependence on sequence length.

Gradient checkpointing reduces retained activations by saving selected boundaries and recomputing intermediate values during the backward pass. That trades extra computation for less saved data. Memory-efficient attention addresses a different allocation by avoiding storage of full token-by-token attention matrices. Both techniques can help, but neither removes model weights, optimizer states or every temporary tensor.

A larger effective batch does not require every sequence to be resident at once. Gradient accumulation processes successive micro-batches before a weight update. Replicated data parallelism processes different micro-batches on multiple GPUs, each holding a complete model replica. Those GPUs increase the number of examples contributing to an update; their memory capacities do not become one shared pool.

A useful memory estimate therefore describes a particular model, training method and execution setup. It supports choosing a configuration to test. Actual peak memory also depends on tensor lifetimes, precision choices, attention kernels and temporary allocations, so memory feasibility alone cannot establish training speed or model quality.

How to Use This Tool:

Match the settings to the training run you intend to execute. Estimates update as you make changes. Calculations use architecture dimensions and numeric settings in your browser; no model weights or training dataset are required.

1. Choose the matching Model preset, or select Custom dense decoder and enter the dimensions from your model's configuration. Custom dimensions assume dense attention projections without biases and a gated feed-forward network.
2. Select the Training method. For LoRA or QLoRA, set Adapter rank and Adapter targets to match the projections you will train.
3. Choose a GPU memory class, or select Custom per-GPU budget and enter the available memory with the correct GB or GiB unit. Preset capacities are nominal GiB planning budgets. Set GPU replicas for replicated training without adding the cards' capacities together.
4. Enter Sequence length and Micro-batch per GPU. Use the sequence length actually processed after padding or packing. Set Gradient accumulation steps under Advanced when several successive micro-batches contribute to each update.
5. Match Trainable precision and Optimizer state to the allocations your trainer uses. Check Gradient checkpointing and Memory-efficient attention against the features actually active in the training run; both are enabled in the opening configuration.
6. Review the per-GPU estimate and Headroom, then use What fits to compare feasible changes before testing them in training.
   If Check training inputs appears, correct the first reported field. For example, a micro-batch of 0 must be replaced with a whole number from 1 to 1,024. Results return when the active inputs are valid.
   Configuration changes are reflected in the page address. A shared link includes those settings.

Interpreting Results:

The headline is an estimated peak for each GPU. Headroom is the selected per-GPU budget minus that estimate. A negative value is the estimated shortfall.

• Method comparison changes the training method while retaining the other assumptions. Adapter rows use the chosen rank and targets. This compares estimated memory, without predicting speed or accuracy.
• Largest micro-batch holds the current sequence length fixed. Longest sequence holds the current micro-batch fixed. Their maxima are separate alternatives and must not be combined as though they were one verified configuration.
• The other What fits alternatives each change one setting from the current configuration. Savings are not cumulative. Rank 8 can increase memory if the current rank is smaller, and an already selected setting may show no change.
• Smallest preset that fits identifies a capacity class only. Its headroom uses that preset's budget. It does not assess compute performance, precision support or interconnects.

Sequence sensitivity shows how estimated memory changes at the current micro-batch and other settings. A curve below the budget line is still subject to the same assumptions.

Before committing to a long run, measure several complete training steps with representative sequence lengths, including the backward pass and optimizer update. Loading the model or completing only a forward pass does not exercise every allocation.

Technical Details:

Parameter storage and temporary training tensors respond to different inputs. Architecture and training method determine the weight and training-state counts. Micro-batch, sequence length, checkpointing and attention behavior determine the estimated activation demand. The following accounting assumes those listed allocations contribute to a concurrent per-GPU planning budget.

Formula Core

The peak estimate adds weights, trainable state, activations, logits and dequantization workspace, then includes a fixed and optional memory allowance.

All terms inside the fraction are bytes. W is base and adapter weights, T is master weights plus gradients and optimizer states, X is saved and active tensors, Z is logits, and J is dequantization workspace. The result is GiB per GPU. E is Extra memory allowance (GiB), from 0 to 128, added to the fixed 2 GiB allowance.

Architecture and parameter counts

The parameter count covers attention projections, a gated multilayer perceptron (MLP), token embeddings, output weights and root mean square normalization (RMSNorm) weights. The architecture symbols correspond to the custom model fields.

The combined query width Q and the width U of each key/value projection follow from the head counts. Grouped-query attention (GQA) shares key/value heads among query heads, so the key and value projections can be smaller than the query projection.

Each decoder block contributes four attention projections and three MLP projections. Embedding and output weights are counted once when tied, or twice when separate.

Here t is 1 for tied embedding/output weights and 2 otherwise. The normalization indicator ε is 1 for the Qwen3 8B preset's additional query/key normalization weights and 0 for the Llama presets and custom architecture. P is the total base parameter count. Custom KV heads must divide the query head count, and head width is entered separately rather than forced to equal hidden width divided by query heads.

For a single projection, a rank-r LoRA pair has r times the sum of its input and output widths in trainable parameters. Summing those pairs across every decoder block gives the following adapter counts.

Attention targets include query, key, value and output projections. The wider target set also includes MLP gate, up and down projections. R denotes the selected adapter count. Full fine-tuning trains P parameters; LoRA and QLoRA train R. Rank is a whole number from 1 to 256 and does not change the frozen base parameter count.

For the Llama 3.1 8B preset, H and Q are 4,096, U is 1,024, and L is 32. At rank 16, the attention-only count can be checked directly.

Including the MLP projections at the same rank raises the count to 41,943,040. These are trainable adapter parameters in addition to the base model.

Weight storage and quantization

Bfloat16 (BF16) uses two bytes per stored value; 32-bit floating point (FP32) uses four. Full fine-tuning and LoRA count every base parameter in BF16. QLoRA applies four-bit NormalFloat (NF4) storage only to the dense linear weights and retains the remaining base tensors in FP32 under the assumed preparation policy.

Double quantization compresses the scale information accompanying the four-bit weights. The coefficient c includes the idealized scale overhead for blocks of 64 weights and groups of 256 scales.

The coefficient is bytes per quantized linear parameter, not bytes per parameter of the entire model. Embeddings, output weights and normalization weights remain at four bytes each in the QLoRA estimate, including when pure BF16 trainable precision is selected. Quantization block rounding and additional casts are not individually counted.

Gradients, master weights and optimizer states

Only trainable parameters receive these allocations. The precision choices are explicit storage assumptions rather than universal rules for every trainer.

Optimizer storage is chosen separately from trainable precision. Changing to 8-bit states changes the moment storage, without quantizing the trainable weights or gradients.

If P_t is the trainable count, and m, g and o are the selected master, gradient and optimizer bytes per trainable parameter, the additional training state is:

The two-byte working weights are already included in W. With the mixed BF16 policy and FP32 AdamW moments, weights and training state together cost 18 bytes per trainable parameter. For adapter methods, this applies to the adapters, with frozen base storage added separately. The nominal 8-bit estimate omits scale metadata and the higher-precision states that some optimizers retain for small tensors.

Activation approximation

Let B be Micro-batch per GPU and S be Sequence length. A one-layer activation baseline follows the accounting in Korthikanti and colleagues' transformer study. The quadratic term represents stored attention intermediates.

Here a is bytes for one active layer and χ is 0 when memory-efficient attention is assumed, or 1 otherwise. Removing this term represents a fused attention kernel that avoids storing full quadratic attention matrices. It does not remove the other activation term. Attention software can fall back to a conventional kernel, so this assumption needs to match the kernel actually used.

Without checkpointing, the baseline is retained for every layer. With full-layer checkpointing, the estimate retains layer boundaries and allows for one recomputed active layer.

LoRA and QLoRA add a small estimate for adapter intermediates. There are n = 4 targeted projections per layer for attention, or n = 7 for attention plus MLP. The retained-layer factor ℓ is 1 with checkpointing and L without it.

Full fine-tuning has no adapter-intermediate term, so X_r is 0. The multiplier α is Activation estimate multiplier, initially 1 and adjustable from 0.25 to 4. It scales saved and active tensors together and does not scale logits.

The coefficient 34 comes from a particular transformer activation analysis. Applying it to these dense decoders, removing the attention term, adding an active recomputed layer and estimating adapter intermediates are approximations. They are not measured peaks or guaranteed upper bounds. In particular, the baseline does not separately adjust for every MLP width or apply an additional GQA activation discount. Unusual custom architectures can differ substantially.

Logits and dequantization workspace

The logits allowance holds one FP32 score for every token and vocabulary entry. QLoRA also allows one largest dense linear weight matrix to be temporarily represented in BF16 for dequantization.

J is 0 for full fine-tuning and LoRA. These workspace assumptions do not establish exact allocation lifetimes: a training setup may avoid a full logits allocation, use different dequantization buffers or need additional temporary copies. The fixed 2 GiB allowance is a planning provision, not a measured hardware constant. Extra allowance can account for observed allocator, communication or optimizer-step overhead.

Effective batch and replicated training

With sequential gradient accumulation, the number of sequences contributing to one update is:

G is Gradient accumulation steps, from 1 to 1,024, and N is GPU replicas, from 1 to 128. Increasing either leaves this per-GPU estimate unchanged at fixed micro-batch and sequence length. This assumes successive micro-batches release their activations and gradient communication reuses gradient storage. Separate communication buffers and first-step allocations can require more memory in a real run.

Units, precision and fit limits

One GiB is 2³⁰ bytes and one GB is 10⁹ bytes. Custom budgets are converted to GiB before comparison, and changing the unit preserves the underlying byte quantity. The accepted custom budget is 1 to 2,048 GiB per GPU. Preset budgets represent nominal capacity classes and do not detect available memory on an installed card.

Headroom uses the selected per-GPU capacity C. Equality belongs to the fitting result.

Calculations retain intermediate precision. Summary estimates use two decimal places and memory components use three; rounding is applied only for display. A displayed 24.00 GiB can therefore lie just above or just below a 24 GiB budget.

The largest fitting micro-batch is searched from 1 to 1,024. Sequence search runs from 1 token to the selected context cap: 131,072 for the Llama presets and custom decoder, or 32,768 for Qwen3 8B. A limit marked with a plus sign has reached the model or search cap; larger values have not been established as valid. If even the minimum tested value exceeds the budget, the corresponding result reports that no micro-batch or sequence fits.

The supported presets are Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B and Qwen3 8B. Custom counts assume the dense decoder structure described above, without projection biases, mixture-of-experts routing or multimodal encoders. Adapter modes keep embeddings and output weights frozen; selectively unfreezing them is not modeled. The estimate excludes sharded weights or optimizer states, tensor and pipeline parallelism, CPU or disk offload, and optimizer paging. It also does not calculate inference KV-cache memory, dataset storage, training duration or rental cost. Padding, loss computation, kernel fallback, fragmentation and differing tensor precision can change measured training memory.

Worked Examples:

References:

• Llama 3.1 8B architecture configuration, Unsloth model mirror.
• Model training anatomy, Hugging Face, version 4.38.0.
• LoRA: Low-Rank Adaptation of Large Language Models, Hu and colleagues, 2021.
• QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers and colleagues, 2023.
• Reducing Activation Recomputation in Large Transformer Models, Korthikanti and colleagues, 2022.
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao and colleagues, 2022.
• Preparation for training quantized models, Hugging Face.
• 8-bit optimizers, Hugging Face.
• Gradient accumulation, Hugging Face.
• Scaled dot product attention and kernel selection, PyTorch documentation.
• Distributed data parallelism and gradient storage, PyTorch documentation.