---
id: "20260711-1552-hop-os-virtual-memory-in-llm-serving"
title: "LLM serving re-imported 1960s virtual-memory theory wholesale — pages, thrashing, and all"
type: "capture"
status: "promoted"
origin: "hop-batch"
writer_model: "claude-opus-4-8"
date_created: "2026-07-11T00:00:00.000Z"
hop_chain: ["SEED: claim-kv-cache-grows-with-context (vault) — hook: KV cache 'paged to slower storage'","KV-cache note -> PagedAttention/vLLM paper, cross-domain bridge: OS virtual memory borrowed for KV cache (max_cosine 0.672, bridge_candidate)","PagedAttention -> Denning's Virtual Memory history, cross-time bridge: the 1959 idea being borrowed (max_cosine 0.683, bridge_candidate)","Denning -> modern LLM-serving 'thrashing', mechanism + road-home: vLLM preemption & llm-d 'cache thrashing' (max_cosine 0.706)","thrashing -> StreamingLLM attention sinks, zoom-in on the causal 'why': locality vs a non-local anchor (max_cosine 0.707)"]
novelty_max_cosine: 0.722
tags: ["LLM","inference","KV-cache","virtual-memory","operating-systems","PagedAttention","thrashing","history-of-computing","borrowed-ideas"]
source_url: "https://arxiv.org/abs/2309.06180"
source_author: "Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica"
source_date: "2023-09-12"
source_tier: 1
source_url_2: "https://denninginstitute.com/pjd/PUBS/ENC/CRC-vm-2013.pdf"
source_author_2: "Peter J. Denning"
source_tier_2: 2
source_url_3: "https://arxiv.org/abs/2309.17453"
source_author_3: "Xiao, Tian, Chen, Han, Lewis"
source_tier_3: 1
source_url_4: "https://llm-d.ai/blog/kvcache-wins-you-can-see"
source_tier_4: 3
promoted_to: ["30-notes/claim-pagedattention-borrows-os-virtual-memory-paging.md","30-notes/claim-atlas-1959-first-virtual-memory-one-level-store.md","30-notes/claim-denning-1968-working-set-model-solves-thrashing.md","30-notes/claim-llm-serving-vllm-reimports-os-preemption-and-thrashing.md","30-notes/claim-streamingllm-attention-sinks-beat-pure-recency-eviction.md"]
not_promoted: ["The 'round trip is a loop' synthesis framing (Seek's commentary paragraph, 'the road home to AI...') — not itself an atomic sourced claim; the underlying facts it synthesizes are each promoted individually above, and the framing survives as connective tissue across those five notes and the new MOC section instead of a standalone note.","'Virtual' is borrowed from optics (virtual image = illusion) — Denning — flagged in the capture itself as an unresearched saved hook (optics->computing bridge), not a followed hop. Left for a future session; no note written on an unverified lead.","Maurice Wilkes' 1965 'slave memory' as ancestor of CPU cache — Denning — same status: saved hook, not followed, candidate for a future mechanism-note once actually hopped.","Ion Stoica as serial-systems-to-AI builder (Spark/Ray/Mesos/vLLM) — PagedAttention author list — saved hook explicitly marked 'Unwritten person-note' in the capture; no independent research done on his career, so no person-note written from a single author-list line."]
promoted_notes_count: 5
promotion_date: "2026-07-12T00:00:00.000Z"
---


The KV-cache seed note mentioned that long contexts may be "paged to slower storage." That word — *paged* — is not a loose metaphor. The 2023 vLLM system made the borrow explicit and the field then re-derived the rest of a 60-year-old operating-systems playbook.

**1. PagedAttention is virtual memory, stated as such.** vLLM proposes "PagedAttention, an attention algorithm inspired by the operating system's (OS) solution to memory fragmentation and sharing: virtual memory with paging"; the mapping is literal — "one can think of blocks as pages, tokens as bytes, and requests as processes" (Kwon et al., SOSP 2023, [arXiv 2309.06180](https://arxiv.org/abs/2309.06180), Tier 1). Before it, systems wasted most KV memory — "only 20.4%–38.2% of the KV cache memory is used to store the actual token states in the existing systems."

**2. The borrowed idea is from 1959.** "The first operating system with virtual memory was the 1959 Manchester Atlas… They called their virtual memory a 'one-level store.'" Multiprogramming it then hit "thrashing… a condition of near-total performance collapse," which engineers called "paging to death," solved by Denning's 1968 working-set model (Denning, *Virtual Memory*, Tier 2).

**3. The round-trip: the vocabulary came back.** Modern serving re-uses it all — vLLM "can preempt requests… recomputed when sufficient KV cache space becomes available," and llm-d names the failure directly: "This instability is caused by 'cache thrashing.'" (Tier 1/3). But recency alone isn't enough: StreamingLLM shows window attention "fails when the text length surpasses the cache size," and "keeping the KV of initial tokens will largely recover the performance" (Xiao et al. 2023, [arXiv 2309.17453](https://arxiv.org/abs/2309.17453), Tier 1).

> [!note] Seek's commentary:
> The road home to AI is a loop: an idea leaves OS theory (1959–1968), sits dormant, and returns intact — same words, same failure mode — as soon as GPUs recreate the 1960s condition of "expensive fast memory, cheap slow memory, too many jobs."

## Why this was hop-worthy
A confirmed cross-domain *and* cross-time bridge (OS 1959 ↔ AI 2023) that links the vault's AI-inference cluster to its history-of-computing / borrowed-ideas cluster.

## Further leads
- The word "virtual" itself is borrowed from optics — a virtual image is "an illusion, made from traces of a real object that is actually somewhere else" (Denning). Optics → computing bridge.
- Maurice Wilkes' 1965 "slave memory" → CPU cache: another descendant of the same address-mapping principle.
- Ion Stoica (vLLM co-author): serial systems→AI builder (Spark, Ray, Mesos). Unwritten person-note.

## Hop chain

### Chain: KV cache → OS virtual memory (1959) → thrashing → attention sinks

Hop 1: Efficient Memory Management for LLM Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- Hook type: Cross-domain bridge (operating systems ↔ LLM serving)
- Hook: seed note's phrase "paged to slower storage" → vLLM's explicit "blocks as pages, tokens as bytes, requests as processes."
- Why followed: vault_bridge returned bridge_candidate=true; highest-priority hook type.
- Key findings: PagedAttention borrows OS virtual memory to stop KV fragmentation (existing systems used only 20–38% of KV memory for real tokens); 2–4× throughput. Tier 1.

Hop 2: Peter Denning, *Virtual Memory* (encyclopedia article) — https://denninginstitute.com/pjd/PUBS/ENC/CRC-vm-2013.pdf
- Hook type: Cross-time bridge (1959 idea inside a 2023 system) + surprising claim
- Hook: "inspired by the classical virtual memory… in operating systems" — where did that come from?
- Why followed: cross-time bridges get extra weight; lands in the vault's precursor/priority cluster.
- Key findings: Atlas 1959 "one-level store"; overlay work once cost programmers "half to two-thirds of their time"; VM was "the subject of intense controversy after its introduction in 1959," settled by Sayre 1969; thrashing ("paging to death") solved by Denning's 1968 working-set model. Tier 2.

Hop 3: vLLM optimization docs + llm-d blog — https://docs.vllm.ai/en/stable/configuration/optimization/ , https://llm-d.ai/blog/kvcache-wins-you-can-see
- Hook type: Mechanism question + road-home to AI
- Hook: does the 1968 word "thrashing" appear in modern LLM serving?
- Why followed: closes the loop — tests whether the borrowed framework brought its failure modes too.
- Key findings: vLLM preempts and recomputes/swaps KV cache; llm-d literally names "cache thrashing" for cache-blind eviction churn. The 1960s vocabulary returned intact. Tier 1/3.

Hop 4: Efficient Streaming Language Models with Attention Sinks — https://arxiv.org/abs/2309.17453
- Hook type: Surprising claim (zoom-in on the causal "why")
- Hook: if eviction works by locality (Denning's working set = recent pages), can you just keep recent tokens?
- Why followed: tests whether the OS analogy holds all the way down to *why* it works.
- Key findings: pure recency ("window attention") "fails when the text length surpasses the cache size"; you must also keep the first tokens ("attention sink"). Not clean locality — a fixed non-local anchor. Tier 1.

Saved hooks not followed:
- The word "virtual" is borrowed from optics (virtual image = illusion) — Denning — an optics→computing bridge, its own capturable cross-domain hook.
- Maurice Wilkes' 1965 "slave memory" as the ancestor of CPU cache — Denning — a second descendant of address-mapping worth a mechanism note.
- Ion Stoica as serial systems→AI builder (Spark/Ray/Mesos/vLLM) — PagedAttention author list — person-note candidate.

Surprise: expected virtual memory to have been embraced instantly as an obvious win — found it was "the subject of intense controversy after its introduction in 1959," debated for a decade until Sayre 1969 settled it.
Surprise: expected KV-cache eviction to rest on Denning-style locality (keep the recent working set) — found pure recency collapses and you must also retain the *first* tokens (attention sinks), a non-local anchor with no clean OS analog.

post-worthy: yes — a tight, verifiable round-trip showing an AI system re-importing a named 1960s OS framework down to its failure modes, with one genuine twist where the analogy breaks.