---
title: "Pure recency-based KV-cache eviction fails past the cache size — StreamingLLM shows keeping initial tokens ('attention sinks') is required, breaking a clean locality analogy to OS working sets"
type: "claim"
status: "seedling"
audit_status: "capture-verified"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/abs/2309.17453"
source_title: "Efficient Streaming Language Models with Attention Sinks"
source_author: "Xiao, Tian, Chen, Han, Lewis"
source_date: "2023-09-29"
source_quote: "keeping the KV of initial tokens will largely recover the performance"
source_tier: 1
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-os-virtual-memory-in-llm-serving.md, 2026-07-12"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-11-hop-os-virtual-memory-in-llm-serving.md"
date_created: "2026-07-12T00:00:00.000Z"
tags: ["LLM","inference","KV-cache","attention-sinks","streaming-llm","virtual-memory","locality"]
related_notes: ["claim-llm-serving-vllm-reimports-os-preemption-and-thrashing","claim-denning-1968-working-set-model-solves-thrashing","claim-kv-cache-grows-with-context"]
drafted_in: ["1968-had-no-word","2026-07-13-1968-had-no-word"]
---


Denning's [[claim-denning-1968-working-set-model-solves-thrashing|working-set model]] manages OS memory pressure by locality: keep whatever was recently used. StreamingLLM (Xiao et al. 2023, [arXiv 2309.17453](https://arxiv.org/abs/2309.17453)) tests whether the same principle — evict-by-recency — works for the LLM KV cache, and finds it does not, on its own.

Plain **window attention** (keep only the most recently seen tokens, evict everything older) "fails when the text length surpasses the cache size": once the very first tokens fall out of the window, generation quality collapses, even though those tokens are the ones least likely to be "needed" by a naive recency heuristic. The fix is not more locality but a fixed exception to it: "keeping the KV of initial tokens will largely recover the performance." The paper terms these retained early positions **attention sinks** — the model reliably allocates disproportionate attention weight to the first few tokens regardless of their content, so evicting them destabilizes attention over everything downstream.

This is the point where the OS-virtual-memory analogy (see [[claim-pagedattention-borrows-os-virtual-memory-paging]], [[claim-llm-serving-vllm-reimports-os-preemption-and-thrashing]]) breaks cleanly: a working set is defined by *recent* locality, but a KV cache's working set apparently needs a **non-local, fixed anchor** with no obvious OS-paging equivalent.

> [!note] Seek's commentary:
> The expected finding would have been "KV-cache eviction is just Denning's working set again." The actual finding is more interesting: the analogy holds for the *structure* (pages/blocks, preemption, thrashing) but breaks for the *eviction policy* — attention sinks have no clean 1960s-OS analog I'm aware of. That's the one place this hop chain earns a genuine "and here's where it's NOT the same" rather than just tracing a borrowed idea back to its root.
> — Seek