LLM KV Cache Calculator

Loading a large language model into GPU memory does not settle how many requests a server can handle at once. Each conversation also needs temporary state as the model reads its prompt and generates an answer. Long documents, accumulated chat history, and lengthy responses can make that request state large enough to limit concurrency even when the model weights fit comfortably.

During autoregressive inference, a model produces tokens one after another. Attention uses a query to select information from earlier tokens through their key and value vectors. The key/value cache, usually shortened to KV cache, retains attention state so later generation steps can reuse it. Conventional attention stores the earlier key and value vectors in each layer. Prompt processing builds the initial cache, and decoding adds state as the answer grows. This trades GPU memory for avoiding repeated computation of the earlier keys and values.

Weights and cached request state have different lifetimes. The learned weights are shared by sequences using the same model instance. Each sequence has its own evolving context and cache unless a serving system explicitly shares reusable portions. A sequence means one retained token stream, so a concurrency estimate is not a count of registered users or requests arriving per second.

Context includes the prompt and generated tokens. Conversation history, system instructions, and retrieved passages all contribute to the prompt. Tokens are model-specific units of text, not a fixed number of words. Planning for the complete prompt plus the maximum intended answer gives a memory budget for a request near its longest allowed length. Live requests can occupy less memory while they are still short.

The attention design determines what must be retained. Model parameter count alone cannot predict KV-cache size, and several of these designs can appear within one model.

Precision changes the storage cost of those retained values. A uniform 8-bit cache payload uses half the bytes of a 16-bit payload, but quantization support, scaling information, and output quality still need checking. Weight precision is a separate choice. Compressing model weights leaves more space for request state without automatically changing the precision of the KV cache.

A serving memory budget also needs room for temporary computation, execution buffers, and allocation overhead. A memory-utilization fraction limits how much device memory the model instance may use across weights, runtime needs, and cached state. It does not describe how busy the GPU processors are. Paged allocation then assigns cache storage in whole token blocks, so the last partly filled block still consumes memory.

Tensor parallelism spreads one model instance across several GPUs. It can divide weights and stored attention heads, but small KV-head counts and compressed latent caches can require replication. Aggregate GPU capacity therefore cannot always be treated as one perfectly divisible memory pool. The resulting memory ceiling describes how many sequences can remain resident under stated assumptions. Acceptable response time, token throughput, and the scheduler's admission decisions require separate measurements with the intended workload.

How to Use This Tool:

Plan one model instance running on a group of identical GPUs. Results update as you change the inputs.

1. Choose Model architecture from the presets or select the custom architecture. Presets use a configuration snapshot dated 26 October 2026. For a custom model, enter its resident parameter count and cached-layer dimensions, using the explicitly declared head dimension when available.
2. Enter Memory per GPU and Tensor-parallel GPUs. GPU presets supply marketed decimal GB; use measured per-device capacity when available. Changing between GB and GiB preserves the amount of memory.
   If Capacity inputs need attention reports incompatible tensor parallelism, choose a count that evenly divides the query heads and can shard or replicate the KV heads. For Llama 3.1 8B, three GPUs fail this check; four GPUs pass it.
3. Enter Tokens per request as the prompt plus the maximum generated answer, then set Target concurrent sequences. The plan budgets that complete token count for every resident sequence.
4. Choose Weight payload precision and KV-cache precision independently, then match Paged allocation block to the intended runtime. The initial 16-token block is a planning assumption, not a universal engine setting.
5. Set GPU memory utilization and refine the budget in Advanced. Add measured runtime reserve, weight metadata overhead, or additional fixed state where needed. The initial utilization is 90%, and all three advanced additions start at zero.
6. Choose the Sliding allocation policy if it appears. Keep Full history for conservative retention planning. Select Ideal eviction only when the deployment reclaims expired window state, with separate allowance for its actual allocation overhead.
7. Review the selected row in Context capacity. If the target exceeds the pool, compare shorter contexts or the candidate counts in GPU sizing. Use a fitting context and sequence count as a starting point for checking the serving engine's actual cache allocation and performance.

Interpreting Results:

The Memory-only concurrency ceiling is the number of complete sequences that fit across the selected tensor-parallel group. Each sequence uses memory on every GPU in that group. Do not multiply the sequence count by the number of GPUs.

Request/GPU includes the rounded KV allocation and any additional fixed state for one sequence. Pool/GPU is the remaining per-device budget after weights and runtime reserve. KV/token/GPU sums the per-token payload across all cached layers; with ideal sliding eviction, it is not the marginal cost of another token after the window fills.

A ceiling of zero is a valid result. Weights and reserve may exhaust the budget, or the remaining pool may be too small for one complete request. Memory derivation shows which quantities consume the budget.

GPU sizing marks the smallest head-compatible count from 1 to 64 that meets the target at the selected context. It also gives the largest context that meets the target at each count, capped by the recorded architecture limit. If no count fits, there is no minimum-for-target marker. Head compatibility and memory fit do not certify hardware, interconnect, or runtime support.

Context pressure compares sampled BF16 and FP8 cache capacities. Its selected point uses your actual cache precision, including FP32 when chosen. Connecting lines do not establish exact capacity between samples; enter the context you intend to serve to obtain its whole-sequence ceiling.

Technical Details:

Every retained sequence consumes state in the model's cached attention layers. The per-GPU demand depends on the stored representation, how many heads reside on that GPU, and how many token blocks remain allocated. Model weights and runtime reserve first reduce the memory available for those sequences.

Formula Core

Let A be the available pool in bytes per GPU and S the bytes required by one sequence on each GPU. Only complete sequences count, so the concurrency ceiling N rounds down.

The estimated weights per GPU are the resident parameter count multiplied by payload bytes and metadata overhead, then divided across the tensor-parallel group.

The pool is the utilization-limited device budget after subtracting weights and extra runtime reserve. An exhausted budget produces a zero pool.

Weight payload uses 2 bytes per parameter for BF16/FP16, 1 for FP8/INT8, and 0.5 for INT4. These are storage assumptions before the supplied metadata overhead. A mixture-of-experts model still needs all resident expert weights counted, even when only some experts run for each token. Rounded parameter estimates, mixed weight formats, and replicated tensors can make actual storage differ from the assumed even division.

The utilization fraction applies before the weight and reserve deductions. Memory outside that fraction remains outside the model-instance budget. The extra reserve accounts for additional runtime needs inside the budget, such as temporary activations, execution buffers, cache scales, and allocator overhead. The initial zero reserve does not imply those costs are absent.

MHA, GQA, and tensor-parallel head allocation

For conventional attention with equal key and value head dimensions, each layer stores two vectors per retained token and local KV head. If h is the KV-head count on one GPU, d is the head dimension, and b is the bytes per cached value, the per-layer cost c is:

The result is bytes per retained token per layer per GPU. The factor of two counts keys and values. BF16/FP16 cache uses 2 bytes per value, FP8 uses 1, and FP32 uses 4. MHA has as many KV heads as query heads; GQA shares fewer KV heads among the query heads. The stored KV-head count therefore determines cache size, while the query-head count also constrains the permitted GPU groups.

Let H_KV be the model's total KV-head count. Under the modeled tensor parallelism, heads divide evenly until each GPU has one. Larger groups replicate those heads.

The GPU count must divide the query-head count. At or below the KV-head count, it must divide the KV heads too. Above that count, the GPU count must be a multiple of the KV-head count. Fractional heads are not allocated.

Llama 3.1 8B illustrates the replication boundary. Its 32 cached layers each use 8 KV heads with a head dimension of 128. For an 8,192-token sequence, BF16 cache and 16-token blocks produce the following allocation without added fixed state:

Moving from eight to sixteen GPUs leaves the cache at 128 MiB per GPU and doubles its aggregate storage. The estimated weight share can still shrink, but the KV cache no longer halves with each doubling of GPUs.

Compressed MLA storage

Compressed MLA retains a joint latent representation of keys and values plus a positional-key part. For latent width r and positional-key width d_R, the per-layer, per-token cost on each GPU is:

The joint latent already represents both keys and values, so no extra factor of two is added. The DeepSeek R1 preset uses a latent width of 512 and a positional-key width of 64, giving 576 stored values per token per layer across 61 layers.

With BF16 cache and 16-token blocks, the cache payload for one 8,192-token sequence is exactly 575,668,224 bytes, or 549 MiB, on each GPU.

That compressed cache is replicated under the plain tensor parallelism modeled here. Increasing the GPU count does not divide the 549 MiB payload, and the model weights must still fit the separate memory budget. Custom architecture entry does not expose a configurable MLA latent width.

Cache estimates assume uniform storage at the chosen precision. Specialized packed MLA formats, higher-precision portions, scale data, and alignment can change actual allocation. A smaller FP8 estimate does not establish runtime support or equivalent model quality.

Retained tokens and whole-block allocation

Let T be tokens per request and B the allocation block in tokens. Full-attention layers allocate T_f token slots by rounding the context up to a whole block.

With full-history retention, sliding layers receive the same rounded token count. Ideal eviction instead caps their retained history at the window W before rounding, giving T_s slots.

For L_f full-attention layers, L_s sliding layers, and additional fixed state F in bytes per sequence per GPU, the complete request allocation is:

Both cached groups share the specified head dimensions and cache precision. For the compressed MLA preset, all cached layers are full attention. Ideal sliding retention models retained payload and ordinary block rounding; extra window slack and padding used to group different attention layers are not derived automatically.

Block boundaries create steps in capacity. With the default Llama setup and 16-token blocks, 8,192 requested tokens occupy 8,192 slots. A request for 8,193 tokens occupies 8,208 slots, increasing its per-GPU cache from 1,024 to 1,026 MiB and reducing the estimated ceiling from 52 to 51 sequences.

Hybrid linear and recurrent state

Linear or recurrent layers can maintain a fixed-size state instead of retaining a separate key/value pair for every past token. A hybrid model still accumulates KV cache in its conventional attention layers. Their cache belongs in the full or sliding layer counts; the separately measured recurrent state belongs in F.

Additional fixed state per sequence per GPU is entered in MiB and should include any required state snapshots. It is added unchanged when cache precision or GPU count changes. A different deployment may require a new measured per-GPU value because state precision, distribution, and snapshot policy can change. Custom entry requires at least one full or sliding cached layer, so a model containing only fixed-state layers is outside its supported calculation.

Units, limits, and numerical precision

One GB contains 1,000,000,000 bytes; one GiB contains 1,073,741,824 bytes. One MiB contains 1,048,576 bytes, and one KiB contains 1,024 bytes. Parameter billions use the decimal convention. Marketed GPU capacity and measured device capacity can therefore produce different budgets even when their displayed numbers look similar.

Token counts, head counts, layer counts, GPU counts, and allocation blocks must be integers. Tensor parallelism is limited to 1 to 64 GPUs, blocks to 1 to 256 tokens, utilization to 1% to 100%, and target concurrency to 1 to 1,000,000 sequences. Requested context must also stay within the selected architecture's recorded limit. Custom query heads must be evenly divisible by KV heads. Blank, malformed, nonfinite, or out-of-range inputs produce a validation error rather than an estimate.

The maximum context at a target is the largest permitted whole-token length whose calculated ceiling still reaches that target. A value of zero means even the smallest supported request cannot meet it with the current budget. Additional memory does not extend the model's recorded context limit or verify a context-extension method.

Arithmetic uses unrounded byte quantities before presentation. Allocated token counts round up, and admitted sequence counts round down. Displayed decimal values use up to three places; those display values are not fed back into the calculation.

The estimate assumes one identical tensor-parallel group and independently budgeted sequences. It does not model pipeline parallelism, context or decode parallelism, weight or cache offload, shared prefixes, speculative decoding, or training state. The model dimensions and limits are preset assumptions or custom entries, not a live inspection of a checkpoint or server.

Privacy Notes:

Capacity arithmetic runs in your browser from the entered planning values; no prompt text or model checkpoint is needed. Planning values can be carried in the shareable page URL, so treat a shared link as a copy of that configuration. URL contents may also appear in browser history and web requests when the link is opened. Displaying the chart can load an external resource.

Worked Examples:

References:

• How caching works, Hugging Face.
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv, 2023.
• Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv, 2023.
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, arXiv, 2020.
• Engine arguments and memory-budget configuration, vLLM.
• Tensor-parallel attention-head allocation, vLLM.
• Compressed MLA cache representation, vLLM.
• Hybrid KV cache manager, vLLM.
• Quantized KV cache, vLLM.
• Llama model architecture specifications, Meta.
• DeepSeek-R1 model configuration, DeepSeek.
• gpt-oss-20b model configuration, OpenAI.