---
title: "The KV cache grows with context length and can balloon to several times the size of the model itself"
type: "claim"
status: "seedling"
audit_status: "flagged (V-006)"
flags: ["[unverified-mechanism — needs primary] KV-cache growth/memory-bandwidth mechanism rests on a Tier-3 vendor blog (quotes verified verbatim against the source, but sources.md requires Tier 1-2 for technical-mechanism claims). Retro-fitted 2026-07-06 per Cali ruling 7 (audit V-006)."]
date_created: "2026-06-04T00:00:00.000Z"
provenance: "Seek research batch, 2026-06-04"
tags: ["LLM","KV-cache","inference","memory","GPU","VRAM","attention"]
source_url: "https://redis.io/blog/prefill-vs-decode/"
source_title: "Prefill vs Decode: LLM Inference Phases Explained"
source_author: "Jim Allen Wallace"
source_date: "2026-04-28"
source_tier: 3
related_notes: ["claim-llm-inference-prefill-decode","claim-ai-inference-means-running-a-model","claim-inference-engineering-emerged-as-specialty"]
drafted_in: ["1968-had-no-word","2026-07-13-1968-had-no-word"]
---


The **key-value cache** (KV cache) is the central memory structure of [[LLM inference]]. It stores the intermediate results of the [[attention mechanism]] — specifically, the key and value vectors computed for every token that the model has seen — so those computations need not be repeated on each decode step.

"Every decode step depends on every prior token, so the model has to remember the full context. That memory is the KV cache. It starts at the size of your prompt and grows by one entry per generated token." (Jim Allen Wallace, Redis, 2026-04-28)

## Why it matters for memory pressure

At production scale, the KV cache is not a marginal structure. "At scale, with long responses across many concurrent requests, the cache can balloon to several times the size of the model itself. Every decode step has to read all of that, which is a big reason decode is memory-bandwidth-bound." (Jim Allen Wallace, Redis, 2026-04-28)

This means that as [[long-context window|context windows]] expand — from 4K tokens in 2022 to 200K and beyond by 2025 — the KV cache grows proportionally, compounding the [[memory bandwidth]] bottleneck in the [[decode phase]].

## What it enables

Without the KV cache, each decode step would require recomputing attention over the *entire* sequence from scratch, making inference complexity quadratic in context length. The KV cache reduces this to approximately linear per token, by caching the attention state built during [[prefill]]. The Pragmatic Engineer describes the KV cache as "the cached results of the attention algorithm, reused between requests to speed up inference."

## Engineering implications

Managing the KV cache is one of the central problems in inference infrastructure:
- **Batching strategy**: Multiple concurrent requests share GPU memory, competing for KV cache space.
- **Cache eviction**: Long-running contexts may need to be paged to slower storage (CPU RAM or disk) and retrieved.
- **Prefix caching**: If many requests share the same system prompt prefix, the KV vectors for that prefix can be cached and reused across requests, reducing both latency and compute.

The KV cache is thus both the mechanism that makes [[autoregressive generation]] tractable and the primary reason that long-context, high-concurrency inference is memory-dominated rather than compute-dominated. See [[claim-llm-inference-prefill-decode]] for the two-phase framing.

See also: [[claim-llm-inference-prefill-decode]], [[claim-inference-engineering-emerged-as-specialty]], [[claim-ai-inference-means-running-a-model]]