LLM VRAM Calculator

A language model that loads for a short chat may run out of memory when asked to handle a long document. Running an open-weight model locally needs space for its learned weights, the attention history retained during generation, and the working memory used by the inference software. The intended conversation length therefore belongs in the memory budget from the start.

Weights are the numerical parameters learned during training. A model described as 8B has roughly eight billion parameters, but its storage requirement also depends on how those numbers are represented. Quantization stores them at lower precision. GGUF, a model-file format used for local inference, can combine different precisions within one file. A format described as 4-bit consequently takes more than exactly four bits per parameter once scaling information and higher-precision tensors are included. Smaller weights can make a larger model practical, with a possible cost to answer quality.

Mixture-of-experts models, abbreviated MoE, add another distinction. Only some expert weights participate in each generated token, yet a fully resident model still needs memory for all its experts. The active parameter count helps explain the work per token. The total parameter count determines how much weight storage must be accommodated across GPU and any permitted CPU memory.

Conversation history consumes a separate key/value cache, usually called the KV cache. It stores keys and values computed for previous tokens so later tokens can reuse them. Context length includes the input and the generated response, measured in tokens rather than words. A document, instructions and existing conversation can occupy much of that allowance before a new answer begins. Reducing weight precision does not automatically reduce KV-cache precision, and reserving a larger context can increase memory use even before that space fills.

The rate of cache growth depends on the model architecture. Full-attention layers retain information across the context. Sliding-window or chunked attention can limit the history stored by local layers. Multi-head latent attention stores a compressed representation, while hybrid linear-attention models combine growing attention caches with fixed recurrent state. Two models with similar parameter counts can therefore have different memory requirements for a long conversation.

Installed memory also needs to be interpreted according to the hardware. A graphics card and an Apple Silicon Mac provide different memory arrangements.

On Apple Silicon, GPU working-set guidance and wired-memory limits can constrain the practical budget. Wired memory must remain resident while in use. The operating system and other applications still need room, so installed capacity alone cannot establish a safe allocation. A runtime allowance must also cover temporary computation buffers and other execution overhead.

Memory capacity and generation speed answer different questions. During single-stream decoding, the model repeatedly reads weights and attention history to produce successive tokens. Memory bandwidth, the rate at which those bytes can be supplied, often limits this stage. CPU offload can accommodate weights that exceed VRAM, but host-memory access and transfers can slow generation. Splitting layers across multiple GPUs adds capacity without automatically multiplying the speed of one sequential response. A memory fit is useful for choosing a configuration; measured performance and acceptable answer quality still need checking in the intended runtime.

How to Use This Tool:

Size one text-generation sequence at the context you intend to reserve. Results update automatically as the inputs change.

1. Choose a Device preset and set Memory per device to your actual configuration. Capacities are entered in GiB. For a custom device, select dedicated or unified memory and enter its memory bandwidth in GB/s.
2. Select a Model preset and Weight quantization. A Custom full-attention model needs total and active parameter counts, attention layers, KV heads and head dimension. Use an appropriate preset for latent or hybrid attention.
3. Enter Context length for the prompt plus response, then choose a supported KV-cache precision. The exact number field accepts contexts beyond the slider's range.
4. Set Advanced values when your observed GPU memory budget or runtime overhead differs from the baseline. Use Sliding-cache allocation to request full-context local storage if your runtime does not keep those caches compact.
   A zero memory-budget, bandwidth or efficiency override keeps the preset assumption. Extra runtime reserve per device is added on every device.
5. Enter Identical dedicated GPUs or Free CPU RAM for weight offload when evaluating those arrangements. CPU offload also needs a realistic CPU memory bandwidth. Unified memory uses one device and no additional offload RAM.
6. Read the fit verdict, then compare Quantization choices or use Context headroom to choose a context target. Apply a different target by editing Context length.
   If Check your sizing inputs replaces the result, correct the field named in its message. Context must be a whole number from 1 to 16,777,216; custom active parameters cannot exceed total parameters, and custom head dimension must be divisible by 32.

Interpreting Results:

The fit verdict describes the selected memory arrangement. It does not verify that a particular runtime supports the model, quantization or cache format.

• GPU fit means the estimated weights, cache, fixed state and runtime allowance fit the GPU budget. Check the remaining margin before treating a close fit as practical.
• CPU offload means some weights need separate CPU RAM. Check Weights on CPU in Memory breakdown and allow for slower generation.
• Does not fit means the stated GPU and offload budgets cannot accommodate the configuration. Reduce the weight footprint or context, or increase the available memory.

GPU headroom subtracts the entire estimated allocation from the GPU budget, including weights that may be offloaded. A successful CPU-offload result can therefore have negative headroom. The context chart also compares the entire allocation with the GPU budget, so its line can remain above that budget when weight offload is viable.

Max native context is the largest context that satisfies the memory and permitted offload rules within a preset's recorded context limit. Zero means no positive context fits. An entered context above the preset limit still receives a memory estimate and a warning; this does not confirm usable context extension. For a custom model, the search instead stops at the general 16,777,216-token calculation ceiling. That ceiling is not a verified native limit for your model.

Comparison rows need consistent assumptions. Models on this device uses the selected ordinary weight format, while native gpt-oss models use MXFP4. When MXFP4 is selected, ordinary models are compared at Q4_K_M. Devices for this model evaluates one device at each listed capacity with its baseline memory budget, bandwidth and efficiency, without CPU offload. It retains the chosen context, weight format, KV precision, local-cache allocation and extra runtime reserve, but does not carry over the other advanced overrides or GPU count.

Bandwidth estimate is an optimistic decode rate, not a benchmark. Unavailable means no rate is offered for a configuration that does not fit.

Memory summaries show up to two decimal places and rates up to one. Memory breakdown and Calculation method provide memory values to six decimal places for closer inspection. Fit decisions use unrounded values, so equal-looking rounded totals can fall on opposite sides of the budget.

Technical Details:

Inference memory combines quantities that behave differently. Weights remain constant for a selected model and weight format. Attention storage depends on context and architecture, recurrent state can remain fixed, and execution needs a separate allowance.

Formula Core

The total allocation is the sum of those four memory requirements.

M is total required memory, W is weight storage, K is reserved KV cache, S is fixed recurrent state and R is runtime allowance. These memory terms use GiB.

One GiB contains 1,073,741,824 bytes. Parameter counts use decimal billions, and bandwidth uses decimal GB/s, with one billion bytes per second in each GB/s. Device capacities sold as GB are interpreted as GiB for these estimates.

Weight storage and effective precision

For ordinary GGUF estimates, the parameter count and effective bits per weight determine the weight allocation.

P is the total text parameter count in billions, and b is the effective number of bits per weight. The factor of eight converts bits to bytes. The following calibrations account for mixed tensor precision and storage overhead. The smaller-model column applies at or below 10 billion parameters; the other column applies above that threshold. F16 and BF16 are two 16-bit floating-point formats with the same storage estimate here.

These values estimate storage; they are not quality scores or exact sizes for every download. Native gpt-oss instead uses recorded MXFP4 GGUF file sizes. Its expert weights and remaining tensors do not all use the same precision. Total file bytes replace the parameter-based storage estimate; active bytes estimate the weight data read for one token.

KV cache, allocation and attention architecture

Cache storage depends on how many numerical elements each attention layer retains for each token. Summing those elements across layers supports different local and global attention layouts.

The sum covers layers that retain an attention cache. For layer l, E is the number of cached elements per token and A is the allocated token count. The value b_KV is bytes per element. Ordinary attention stores both keys and values, so E is twice the number of KV heads multiplied by head dimension, the number of values in each key or value vector. Grouped-query attention uses the KV-head count here, even when it has more query heads.

Ordinary keys and values use the selected cache precision. F16 stores each value as a 16-bit floating-point number. The quantized formats also store block scales, which explain why an 8-bit cache takes slightly more than one byte per element.

Allocation rounds the requested context T upward to a multiple of 256 tokens. Define p as that padded length.

Full-attention layers reserve p(T) positions. The compact allocation assumes Flash Attention, one sequence and a 512-token micro-batch, meaning up to 512 prompt tokens processed together. For a sliding window or attention chunk of w tokens, local layers reserve the following length.

The additional 512 positions are an allocation allowance. Full-context local allocation instead reserves p(T) positions on local layers too. Global layers continue to use the full padded context, and their KV-head counts and head dimensions may differ from those of local layers.

Multi-head latent attention, or MLA, replaces expanded per-head keys and values with a compressed latent cache. The DeepSeek R1 671B estimate retains 512 latent elements plus 64 positional-key elements per token per layer. The selected cache precision applies once to these 576 elements, without multiplying by the head count or adding a second value cache. This assumes a runtime that uses the compressed MLA cache; expanded keys and values need more memory. Its distilled Qwen-based relatives use full attention instead. Cache compression therefore follows the actual architecture, not the model-family name.

Hybrid linear attention adds fixed recurrent and convolution state. Only its full-attention layers contribute a growing KV cache. These estimates reserve fixed state as F32, meaning four-byte, 32-bit floating-point values, regardless of KV precision.

L_linear counts linear-attention layers, N_recurrent is recurrent-state elements per layer, D_conv is convolution width and k is convolution kernel length. Quantizing the attention cache does not reduce this fixed state. Different runtime allocation policies or unsupported compact caches can require more memory than these assumptions.

Usable capacity and runtime allowance

The GPU budget applies a usable fraction to each device's capacity. Runtime memory is then included in the required allocation, with a reserve on every device.

U is usable GPU capacity in GiB, C is memory per device in GiB, f is the usable fraction and n is device count. The additional reserve r is GiB per device, above the baseline 1 GiB. These are the default fractions before an explicit budget override.

The Apple fractions are conservative historical planning assumptions. They are not universal limits for current macOS versions. The other unified-memory percentages are also sizing defaults; 87.5% is a reserve assumption, not a vendor guarantee. An observed GPU working-set or available-memory budget is a better basis for an override. Apple's recommended working-set value is itself guidance about maintaining performance, rather than proof that every allocation below it will succeed.

Multiple dedicated GPUs are treated as identical cards with a balanced, sequential distribution of layers. Combined capacity grows with device count, and so does runtime reserve. Actual placement must still fit each card; the aggregate estimate does not validate individual layer assignments.

Fit boundaries and weight offload

Headroom compares the entire requirement with usable GPU capacity.

When cache, fixed state and runtime already fit on the GPUs, any remaining shortfall consists of weights that would need CPU RAM. That required offload O is calculated as follows.

Apply these conditions in order. CPU offload is available only for dedicated GPUs with separate available system RAM.

Equality is accepted at both memory boundaries. It leaves no additional margin beyond the runtime reserve already included. Adding CPU RAM cannot fix a shortage caused by GPU-resident cache or runtime memory, and unified memory cannot be counted again as a separate CPU-offload pool.

Single-stream decode bandwidth

Dividing memory traffic by bandwidth estimates the time needed to supply one token's data. With CPU offload, the GPU and CPU access times are added. The inverse of that time, adjusted by an efficiency factor, gives the estimated decode rate.

v is estimated tokens per second. D_GPU and D_CPU are bytes accessed per generated token, while the corresponding B values are memory bandwidths in GB/s. The efficiency factor η defaults to 0.725, or 0.50 for Apple Ultra presets. These are heuristic factors, not measured utilization for the selected run.

GPU traffic includes active weight reads, logical KV reads and twice the fixed-state size, allowing one state read and one write. Logical KV reads use the actual context rather than padded allocation; local attention reads at most its window or chunk length, even when full-context storage is reserved.

For ordinary models, active weight traffic scales with the ratio of active to total parameters. The native gpt-oss active-byte estimates above instead count non-expert weights plus the selected expert tensors. With weight offload, the active traffic assigned to CPU is proportional to the fraction of total weights offloaded.

GPU bandwidth is not multiplied by the number of cards in a sequential layer split. Computation, dequantization, CPU/GPU transfers, interconnect delays and expert routing can impose additional limits that the bandwidth equation does not represent. Those omissions make the estimate particularly optimistic for offloaded and MoE configurations.

Limitations and Privacy:

• Model and hardware presets use a fixed source snapshot dated 26 October 2026. They do not measure your device or perform live specification lookups.
• Vision and audio model parts are excluded. Training, fine-tuning and concurrent multi-user serving require different memory and workload assumptions.
• Prompt-processing speed and time to first token are not modeled. The decode estimate also does not score answer quality or the quality loss from quantization.
• Actual weight residency, temporary buffers, cache support and operating-system memory pressure can change the allocation. A listed model, device or cache format is not a compatibility guarantee.

The sizing calculation runs in your browser without downloading or executing a language model. Input settings can be carried in the page URL, so a shared link can disclose the configuration. The context chart may require a network connection to load its display resources; local calculation does not imply that the entire page makes no network requests.

Worked Examples:

References:

• Sliding-window cache allocation, llama.cpp, revision d854d98.
• Cache strategies, Hugging Face.
• Quantized block-format definitions, GGML.
• Llama 3.1 8B Instruct model configuration, Unsloth AI.
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, DeepSeek-AI, 19 June 2024.
• Gated Delta Networks: Improving Mamba2 with Delta Rule, Yang, Kautz and Hatamizadeh, 6 March 2025.
• Introducing gpt-oss, OpenAI, 5 August 2025.
• Native gpt-oss-20b GGUF weights, GGML.
• Native gpt-oss-120b GGUF weights, GGML.
• Recommended GPU working-set size, Apple Developer Documentation.
• MacBook Pro (16-inch, 2024) technical specifications, Apple.
• Mastering LLM Techniques: Inference Optimization, NVIDIA, 17 November 2023.
• Apple Silicon generation speeds and memory bandwidth, llama.cpp community discussion.
• How to set GPU offload layers in llama.cpp, Simplified Guide.