The KV cache grows with context length and can balloon to several times the size of the model itself
The key-value cache (KV cache) is the central memory structure of LLM inference. It stores the intermediate results of the attention mechanism — specifically, the key and value vectors computed for every token that the model has seen — so those computations need not be repeated on each decode step.
"Every decode step depends on every prior token, so the model has to remember the full context. That memory is the KV cache. It starts at the size of your prompt and grows by one entry per generated token." (Jim Allen Wallace, Redis, 2026-04-28)
Why it matters for memory pressure
At production scale, the KV cache is not a marginal structure. "At scale, with long responses across many concurrent requests, the cache can balloon to several times the size of the model itself. Every decode step has to read all of that, which is a big reason decode is memory-bandwidth-bound." (Jim Allen Wallace, Redis, 2026-04-28)
This means that as context windows expand — from 4K tokens in 2022 to 200K and beyond by 2025 — the KV cache grows proportionally, compounding the memory bandwidth bottleneck in the decode phase.
What it enables
Without the KV cache, each decode step would require recomputing attention over the entire sequence from scratch, making inference complexity quadratic in context length. The KV cache reduces this to approximately linear per token, by caching the attention state built during prefill. The Pragmatic Engineer describes the KV cache as "the cached results of the attention algorithm, reused between requests to speed up inference."
Engineering implications
Managing the KV cache is one of the central problems in inference infrastructure:
- Batching strategy: Multiple concurrent requests share GPU memory, competing for KV cache space.
- Cache eviction: Long-running contexts may need to be paged to slower storage (CPU RAM or disk) and retrieved.
- Prefix caching: If many requests share the same system prompt prefix, the KV vectors for that prefix can be cached and reused across requests, reducing both latency and compute.
The KV cache is thus both the mechanism that makes autoregressive generation tractable and the primary reason that long-context, high-concurrency inference is memory-dominated rather than compute-dominated. See claim-llm-inference-prefill-decode for the two-phase framing.
See also: claim-llm-inference-prefill-decode, claim-inference-engineering-emerged-as-specialty, claim-ai-inference-means-running-a-model