The KV cache is the part of a running model that grows while you use it, and it is usually the reason a GPU that comfortably holds a model's weights still runs out of memory under real traffic. The paper that introduced vLLM's PagedAttention put a number on the problem early: the cache for a single OPT-13B request could reach 1.6 GB at 2,048 tokens, and it grows and shrinks throughout a request in ways ordinary memory allocators handle badly. Everything below is calculated from published model configs, not measured on a live rental.
The formula
For a standard transformer layer, the cache cost per token is:
KV cache bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_element
The 2 accounts for storing both the key and the value. Llama 3.1 8B makes a clean worked example: 32 layers, 8 KV heads, a head dimension of 128, and 2 bytes per element in BF16. That comes to 128 KiB per token, 4.00 GiB for a 32,000-token conversation and 16.00 GiB for a 128,000-token one, calculated from its config.json. Double the context and the cache doubles with it, independent of how many parameters the model has.
GQA shrinks the multiplier before the formula runs
Llama 3.1 8B already benefits from grouped-query attention: it has 32 query heads but only 8 KV heads, because GQA lets several query heads share one key and value projection. Without that trick, an old-style model with 32 KV heads at the same size would cost 512 KiB per token, four times as much, purely from the head count. Most current open models use GQA for this reason, and the KV heads count in a model's config, not the attention heads count, is the number that belongs in the formula above.
Sliding-window layers cap the cost instead of shrinking it
Some models bound each layer's cache to a fixed window of recent tokens rather than the whole conversation. gpt-oss-120b splits its 36 layers evenly: 18 use full attention and grow with every token, and 18 use a 128-token sliding window that stops growing once it fills. Only the 18 full layers scale with context, at 36 KiB per token, so a 128,000-token conversation still costs about 4.5 GiB rather than the 9 GiB a fully-growing model of the same shape would need. Gemma 4 pushes this further: 50 of the 31B model's 60 layers use a 1,024-token sliding window, and their combined cost never passes about 800 MiB, however long the chat runs. The gpt-oss and Gemma 4 pages carry the full per-model numbers.
Linear-attention and state-space layers do not grow at all
A newer class of layer keeps a fixed-size recurrent state per sequence instead of a cache that grows with tokens. Qwen3.8-27B is built from 16 repeated blocks of three Gated DeltaNet layers and one full-attention layer, and only the full-attention layers, 16 of 64 in total, cost anything that scales with context: 64 KiB per token, 2 GiB for a 32,000-token conversation. The 48 DeltaNet layers add a fixed state that this figure does not include. vLLM's design documentation for its hybrid KV cache manager explains why this needs different bookkeeping than a plain cache: each layer type gets its own allocation inside one shared memory pool, with every token kept for full-attention layers but only the window for sliding layers. Mamba-2 layers in the Nemotron family and KDA layers in Kimi K3 work the same way.
MLA compresses the cache instead of shrinking head count
DeepSeek's multi-head latent attention takes a different approach: rather than storing full key and value vectors, it caches a compressed latent representation sized at the model's kv_lora_rank plus qk_rope_head_dim, per layer. Mistral Small 4 uses the same technique and costs 22.5 KiB per token, 0.70 GiB for a 32,000-token conversation, well under what its layer count would suggest without the compression. GLM-5.3 combines MLA with a sparse attention indexer and costs 87.8 KiB per token before counting the indexer's own keys. Kimi K3 and GLM-5.3-Flash use variants of the same idea.
FP8 halves every number above
Setting vLLM's --kv-cache-dtype fp8 stores each cache element in 1 byte instead of 2, which halves every figure on this page at the cost of some accuracy. Support is not universal: vLLM's recipes note that Hopper GPUs cannot use an FP8 KV cache for gpt-oss or for GLM-5.3-Flash, so check the specific model's recipe before planning around it rather than assuming it always applies.
What vLLM actually gives the cache
None of this happens in isolation from the rest of the GPU. vLLM's gpu_memory_utilization setting reserves a fraction of the card for the model executor, weights included, and defaults to 0.92 on the current main branch, 0.9 in older releases, so state the version in any plan that depends on the exact figure. What is left after loading weights, activation memory and CUDA graphs is what the KV cache gets. On one 80 GB GPU running gpt-oss-120b, that works out to roughly 8 GB once the 65.2 GB checkpoint is loaded, room for something like seven 32,000-token conversations at once by the 36 KiB-per-token figure above, before activation memory takes its share. --max-model-len bounds how much any single request can reserve, so lowering it is often the fastest way to raise how many requests run at once on a card that is short on room. The engine prints the real KV cache size and maximum concurrency when it starts, and that number is always more trustworthy than a calculation like this one. My habit is to treat every figure on this page as a sanity check before launching a GPU, then trust the engine's own log over the spreadsheet the moment it disagrees.
Frequently asked questions
Why does a model that fits in memory still run out of room with real traffic?
The weights are a fixed cost, but the KV cache grows with context length and with how many requests run at once. A GPU with just enough room for the weights often has little left for the cache once real conversations start.
Does a bigger context window always mean a bigger KV cache?
For layers that use full attention, yes, linearly. Sliding-window layers cap out at a fixed size regardless of context length, and linear-attention or state-space layers do not grow with context at all, which is why hybrid models cost less than the formula alone would suggest.
What is the fastest way to fit more concurrent users on the same GPU?
Lower max-model-len so each request reserves less cache, switch to an FP8 KV cache where the model and GPU support it, or move to a model with more sliding-window or linear-attention layers relative to full attention.
Is the KV cache the same thing as GPU memory bandwidth?
No. Bandwidth affects how fast tokens are read from the cache during generation. The KV cache itself is a capacity question: how much memory the running conversations occupy at once.
Where do I find the real KV cache size for my GPU, not a calculated one?
In the serving engine's own startup log. vLLM prints the KV cache size and maximum concurrency it settled on for the specific GPU and model as soon as it finishes loading.