The KV cache grows with context length and can balloon to several times the size of the model itself
The key-value cache (KV cache) is the central memory structure of LLM inference. It stores the intermediate results of the attention mechanism — specifically, the key and value vectors computed for every token that the model has seen — so those computations need not be repeated on each decode step.
"Every decode step depends on every prior token, so the model has to remember the full context. That memory is the KV cache. It starts at the size of your prompt and grows by one entry per generated token." (Jim Allen Wallace, Redis, 2026-04-28)
Why it matters for memory pressure
At production scale, the KV cache is not a marginal structure. "At scale, with long responses across many concurrent requests, the cache can balloon to several times the size of the model itself. Every decode step has to read all of that, which is a big reason decode is memory-bandwidth-bound." (Jim Allen Wallace, Redis, 2026-04-28)
This means that as context windows expand — from 4K tokens in 2022 to 200K and beyond by 2025 — the KV cache grows proportionally, compounding the memory bandwidth bottleneck in the decode phase.
What it enables
Without the KV cache, each decode step would require recomputing attention over the entire sequence from scratch, making inference complexity quadratic in context length. The KV cache reduces this to approximately linear per token, by caching the attention state built during prefill. The Pragmatic Engineer describes the KV cache as "the cached results of the attention algorithm, reused between requests to speed up inference." [unverified-quote — needs direct read: no URL recorded, and the phrase was not found at the candidate article (Orosz & Morikawa, 2024-02-20) on 2026-09-11]
Primary corroboration (added 2026-09-11 audit)
The vendor-blog claim has a Tier-1 antecedent. Pope et al. (Google, arXiv:2211.05102, 2022), analysing PaLM inference on TPU v4, state in §2.1: "For a 500B+ model with multihead attention, the attention KV cache grows large: for batch size 512 and context length 2048, the KV cache totals 3TB, which is 3 times the size of the model's parameters. The on-chip memory needs to load this KV cache from off-chip memory once for every token generated during which the computational core of the chip is essentially idle." The same section gives the crossover: "At small batch sizes and sequence lengths, the time to load weights dominates. At larger batch sizes and sequence lengths (e.g. 2048+ tokens with batch size 512+), the time to load the KV cache dominates." That is the growth-past-model-size claim and the memory-bandwidth mechanism in the primary literature, four years before the Redis post.
Engineering implications
Managing the KV cache is one of the central problems in inference infrastructure:
- Batching strategy: Multiple concurrent requests share GPU memory, competing for KV cache space.
- Cache eviction: Long-running contexts may need to be paged to slower storage (CPU RAM or disk) and retrieved.
- Prefix caching: If many requests share the same system prompt prefix, the KV vectors for that prefix can be cached and reused across requests, reducing both latency and compute.
The KV cache is thus both the mechanism that makes autoregressive generation tractable and the primary reason that long-context, high-concurrency inference is memory-dominated rather than compute-dominated. See claim-llm-inference-prefill-decode for the two-phase framing.
See also: claim-llm-inference-prefill-decode, claim-inference-engineering-emerged-as-specialty, claim-ai-inference-means-running-a-model