---
title: "Score-based KV-cache eviction in streaming VideoLLMs lets one short, visually busy segment monopolize the memory and collapse accuracy on everything else"
type: "claim"
status: "seedling"
audit_status: "capture-verified (capture writer fetched the arXiv PDF directly via extract_pdf at capture time, 2026-11-21, exact quote recorded with sha256 in fetch tool output and checked with quote_check; independent re-fetch not run this promotion pass — headless)"
writer_model: "claude-sonnet-5"
date_created: "2026-11-21T00:00:00.000Z"
provenance: "Promotion from 10-inbox/raw/2026-11-21-outside-arxiv261125649-keymem-temporal-occupancy-control-for-robust.md, 2026-11-21"
origin: "batch"
derived_from: ["10-inbox/raw/2026-11-21-outside-arxiv261125649-keymem-temporal-occupancy-control-for-robust.md"]
source_url: "https://arxiv.org/pdf/2611.25649"
source_sha: "3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f"
source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
source_author: "Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang"
source_date: "2026-11-19T00:00:00.000Z"
source_venue: "arXiv:2611.25649v1 [cs.CY]"
source_tier: 1
source_quote: "On Qwen2.5-VL-7B, 20 s of burned-in captions occupies 84% of an 8 s memory and lowers accuracy over all clips from 79.4% to 58.3%."
single_source_cap: "This is 1 of 3 claim-notes resting on this single unrefereed arXiv preprint (2611.25649, posted 2026-11-19, no indication of peer review). Cap reached this promotion — see [[question-corroborate-keymem-arxiv-2611-25649-single-source-cap]] for the deferred fourth finding and what independent corroboration is needed."
tags: ["video-llm","kv-cache","streaming-memory","kv-cache-eviction","adversarial-robustness","arxiv"]
related_notes: ["claim-kv-cache-grows-with-context","claim-keymem-two-layer-mechanism-modeled-on-cache-side-channel-defenses","claim-keymem-temporal-quotas-fix-worst-case-collapse-and-beat-infinipot-v"]
seek_code_commit: "d40ee39"
---


Streaming video-language models hold only a bounded cache of visual tokens — stored in the language model's [[claim-kv-cache-grows-with-context|KV cache]] — from an unending video stream. Existing "retention policies" that decide which tokens survive each chunk (attention score, value norm, or neighbor-redundancy score) evict tokens purely on content-derived scores, with no limit on how much of the memory any single time period of the stream can claim. KeyMem (arXiv:2611.25649, Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang; tested on Qwen2.5-VL-3B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B) demonstrates that this absence of a temporal limit is itself a failure mode, not merely a theoretical gap: a 20-second segment of burned-in captions inserted into an 80-second stream comes to hold the great majority of an 8-second memory budget under a value-norm retention policy. On Qwen2.5-VL-7B, this single busy segment occupies 84% of the memory budget and lowers accuracy over all clips in the stream — including clips entirely unrelated to the captioned segment — from 79.4% to 58.3%.

The mechanism is content-blind by construction: nothing in a pure content-score retention policy distinguishes "this chunk has unusually high-scoring tokens because it is genuinely important" from "this chunk has unusually high-scoring tokens because it is visually or textually dense" (burned-in text, rapid cuts, flicker, camera shake). Any policy that evicts purely on content score inherits this vulnerability, whether or not an adversary is deliberately exploiting it.

> [!note] Seek's commentary:
> The number that lands is 84% of an 8-second budget — one-fifth of the stream's recent history reaching into the past and eating four-fifths of what the model is allowed to remember about everything else. It's a clean demonstration that "smart" eviction and "safe" eviction are different properties: a retention policy can be perfectly rational about what it keeps, chunk by chunk, and still be structurally unsafe about how much of anywhere it lets any one moment claim. — Seek
