Score-based KV-cache eviction in streaming VideoLLMs lets one short, visually busy segment monopolize the memory and collapse accuracy on everything else
Streaming video-language models hold only a bounded cache of visual tokens — stored in the language model's KV cache — from an unending video stream. Existing "retention policies" that decide which tokens survive each chunk (attention score, value norm, or neighbor-redundancy score) evict tokens purely on content-derived scores, with no limit on how much of the memory any single time period of the stream can claim. KeyMem (arXiv:2611.25649, Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang; tested on Qwen2.5-VL-3B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B) demonstrates that this absence of a temporal limit is itself a failure mode, not merely a theoretical gap: a 20-second segment of burned-in captions inserted into an 80-second stream comes to hold the great majority of an 8-second memory budget under a value-norm retention policy. On Qwen2.5-VL-7B, this single busy segment occupies 84% of the memory budget and lowers accuracy over all clips in the stream — including clips entirely unrelated to the captioned segment — from 79.4% to 58.3%.
The mechanism is content-blind by construction: nothing in a pure content-score retention policy distinguishes "this chunk has unusually high-scoring tokens because it is genuinely important" from "this chunk has unusually high-scoring tokens because it is visually or textually dense" (burned-in text, rapid cuts, flicker, camera shake). Any policy that evicts purely on content score inherits this vulnerability, whether or not an adversary is deliberately exploiting it.
Source
“On Qwen2.5-VL-7B, 20 s of burned-in captions occupies 84% of an 8 s memory and lowers accuracy over all clips from 79.4% to 58.3%.”
claude-sonnet-5 · Promotion from 10-inbox/raw/2026-11-21-outside-arxiv261125649-keymem-temporal-occupancy-control-for-robust.md, 2026-11-21 · raw markdown