all tags

KV-cache

7 notes carry this tag — 5 claims, 1 capture, 1 essay. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

Essays

• 
  The one thing 1968 didn't have a word for
  essay draft 2026-07-13 · LLM, inference, KV-cache, virtual-memory, operating-systems

Everything else

• Modern LLM-serving systems reimport OS memory-pressure vocabulary intact — vLLM preempts and recomputes KV cache, and llm-d names cache-eviction churn 'cache thrashing'
  claim seedling Tier 1 2026-07-12 · LLM, inference, KV-cache, vLLM, llm-d
• PagedAttention manages the LLM KV cache with a literal port of OS virtual-memory paging — blocks as pages, tokens as bytes, requests as processes
  claim seedling Tier 1 2026-07-12 · LLM, inference, KV-cache, PagedAttention, vLLM
• Pure recency-based KV-cache eviction fails past the cache size — StreamingLLM shows keeping initial tokens ('attention sinks') is required, breaking a clean locality analogy to OS working sets
  claim seedling Tier 1 2026-07-12 · LLM, inference, KV-cache, attention-sinks, streaming-llm
• LLM serving re-imported 1960s virtual-memory theory wholesale — pages, thrashing, and all
  capture promoted Tier 1 2026-07-11 · LLM, inference, KV-cache, virtual-memory, operating-systems
• The KV cache grows with context length and can balloon to several times the size of the model itself
  claim seedling Tier 3 2026-06-04 · LLM, KV-cache, inference, memory, GPU
• LLM inference splits into two distinct compute phases: prefill (compute-bound) and decode (memory-bandwidth-bound)
  claim seedling Tier 3 2026-06-04 · LLM, inference, prefill, decode, KV-cache