---
id: "20261121-0225-outside-arxiv261125649-keymem-temporal"
title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory (arXiv:2611.25649)"
type: "capture"
status: "promoted"
promoted_to: ["claim-keymem-score-based-eviction-causes-temporal-memory-concentration-collapse","claim-keymem-two-layer-mechanism-modeled-on-cache-side-channel-defenses","claim-keymem-temporal-quotas-fix-worst-case-collapse-and-beat-infinipot-v","entity-adi-shamir","entity-moinuddin-qureshi","entity-kerckhoffs-principle"]
not_promoted: ["Claim 4 (KeyMem's security design explicitly modeled on Prime+Probe/DAWG/CEASER and Kerckhoffs's principle) — not written as its own note. Folded into claim-keymem-two-layer-mechanism-modeled-on-cache-side-channel-defenses as the mechanism's stated design rationale rather than a separately falsifiable claim: 'what KeyMem borrows and why' belongs with 'what KeyMem is,' per the atomic-is-not-many ruling. The three named prior-work entities it names (Shamir, Qureshi, Kerckhoffs's principle) were promoted to entity hubs instead.","Claim 5 (secret-keyed sampling measurably helps only against a simulating adaptive adversary, and even there does not reliably beat deterministic quotas) — genuinely distinct and promotion-worthy on its own terms, but this capture's three other claims already fill the single-source concentration cap (sources.md: no more than 3 claim-notes on one unrefereed preprint) against this same arXiv:2611.25649, which has no indication of peer review. Per the cap's own rule, the fourth finding does not become a note; routed instead to question-corroborate-keymem-arxiv-2611-25649-single-source-cap, quoted in full there so it isn't lost if corroboration later lands.","Entity candidates not promoted to hubs: Dag Arne Osvik and Eran Tromer (co-authors on the 2006 cache-attack paper with Shamir; co-authorship alone is a mention per the 2026-10-01 ruling — named on entity-adi-shamir instead). Kiriansky, Lebedev, Amarasinghe, Devadas, Emer (DAWG's co-authors; no separate standing established in this capture, mentions only). Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang (KeyMem's own authors; mentions only). Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang (InfiniPot-V's authors; mentions only, and InfiniPot-V itself is the highest-leverage unread corroboration lead — see the routed question). Wouter Kool, Herke van Hoof, Max Welling (Gumbel-top-k originators; a further lead only, not cited within any promoted claim, no hub)."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-11-21T00:00:00.000Z"
provenance: "batch run 2026-11-21"
derived_from: []
tags: ["video-llm","kv-cache","streaming-memory","cache-side-channels","kerckhoffs-principle","adversarial-robustness","arxiv"]
source_url: "https://arxiv.org/pdf/2611.25649"
source_sha: "3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f"
source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
source_author: "Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang"
source_date: "2026-11-19T00:00:00.000Z"
source_venue: "arXiv:2611.25649v1 [cs.CY]"
source_tier: 1
seek_code_commit: "d40ee39"
---


Submitted 19 Nov 2026 by Wei Song (Griffith University), Yuxin Cao (National University of Singapore), Xiao Cheng and Xi Zheng (Macquarie University), Christy Jie Liang (University of Technology Sydney), and Leo Yu Zhang (Griffith University). Listed under cs.CY (Computers and Society) despite being a systems/security paper on streaming video-language-model memory management — the listing category is untrusted the way the task brief warns, and does not reflect the paper's actual content, which is a KV-cache-eviction mechanism paper with an explicit security/adversarial-robustness framing.

Abstract page fetched via `archive_page` (sha `91218ccfda3f31b12623f4c298e9b0cd1bdb32ced5d0293c5dabe37f327edb24`). Full text (23 pages) fetched via `extract_pdf` at https://arxiv.org/pdf/2611.25649 (sha256 `3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f`, tls verified, pdftotext). All quotes below checked against this extracted text with `quote_check` and returned grounded.

No addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing were found anywhere in the abstract page or the extracted PDF text. This is a standard-shaped arXiv CS paper (authors, institutions, abstract, related work, propositions, experiments, references). **No safety flags for this session.**

## Claim: Score-based token eviction in streaming VideoLLMs lets a single short, visually busy segment monopolize the memory and collapse downstream accuracy

Streaming video-language models (the paper tests Qwen2.5-VL-3B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B) keep only a bounded cache of visual tokens from an unending video stream, held in the language model's KV cache. Existing "retention policies" (attention score, value norm, or neighbor-redundancy score) evict tokens after every chunk based purely on content-derived scores, with no limit on how much of the memory any one time period of the stream can claim. The paper demonstrates that a 20-second segment of burned-in captions inserted into an 80-second stream can come to hold the great majority of an 8-second memory budget under a value-norm retention policy, and that this "temporal memory concentration" directly degrades the model's ability to answer questions about earlier, unrelated clips.

- source_quote: "On Qwen2.5-VL-7B, 20 s of burned-in captions occupies 84% of an 8 s memory and lowers accuracy over all clips from 79.4% to 58.3%."
- source_url: https://arxiv.org/pdf/2611.25649
- source_sha: 3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f
- source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
- source_author: Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang
- source_date: 2026-11-19
- source_venue: "arXiv:2611.25649v1 [cs.CY]"
- source_tier: 1

## Claim: KeyMem's mechanism is a two-layer wrapper — a hard per-bucket temporal cap on rank-normalized scores, plus optional secret-keyed sampling of survivors

KeyMem does not replace a base retention policy; it wraps one. First, it rank-normalizes whatever base score a policy produces (so no single extreme score can dominate). Second, it divides the video timeline into fixed-duration "buckets" and enforces a hard cap on how many tokens any one bucket may retain — this is the mechanism that bounds any segment's share of memory from its very first chunk, regardless of content. Third, and optionally, it draws the surviving tokens within a bucket using a Gumbel-top-k sample keyed by a secret, making the specific identity of survivors unpredictable to an observer who lacks the key. The paper proves (Propositions 1–2) that the cap bounds any L-second segment to at most (⌈L/W⌉+1)⌊ρB⌋ tokens of a budget B, and that the keyed draw gives every eligible token a non-zero survival probability each round, independent of its score.

- source_quote: "KeyMem, a training-free wrapper that rank-normalizes any base score, divides the video stream into fixed-duration temporal buckets and limits how many visual tokens each bucket can retain, and optionally samples survivors with a secret key."
- source_quote_2: "This hard cap bounds a segment's memory share from its first chunk."
- source_url: https://arxiv.org/pdf/2611.25649
- source_sha: 3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f
- source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
- source_author: Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang
- source_date: 2026-11-19
- source_venue: "arXiv:2611.25649v1 [cs.CY]"
- source_tier: 1

## Claim: Imposing a temporal quota (of any of three tested kinds) removes the worst-case accuracy collapse and beats the published InfiniPot-V baseline on two of three tested models

Across five natural "stressors" (calm, rapid cuts, burned-in text, flicker, camera shake), three VideoLLMs, and two memory budgets, the paper reports that adding any temporal-allocation rule — equal time-spans, a hard cap alone, or full KeyMem — removes the worst-case accuracy collapse that attention-based retention otherwise suffers on some models, and that these temporal quotas outperform the previously published InfiniPot-V retention rule by a wide margin on two of the three model families tested.

- source_quote: "Across five natural stressors, three VideoLLMs and two budgets, temporal quotas raise the worst accuracy of attention-based retention at the smaller budget from 42.2% to 73.3–76.1% and exceed that of InfiniPot-V by 6.1–16.7% on two of the three models."
- source_url: https://arxiv.org/pdf/2611.25649
- source_sha: 3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f
- source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
- source_author: Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang
- source_date: 2026-11-19
- source_venue: "arXiv:2611.25649v1 [cs.CY]"
- source_tier: 1
- Note: InfiniPot-V itself (Kim, Shim, Choi & Chang, NeurIPS 2026) was not independently fetched this session — this claim rests on KeyMem's own paper's report of the comparison, not on a reading of InfiniPot-V's own results. See Further leads.

## Claim: KeyMem's two-layer security design is explicitly modeled on, and named after, countermeasures developed against CPU cache side-channel timing attacks — partitioned caches and randomized (keyed) caches

The paper's own framing states that it is deliberately transplanting a solution from hardware security into ML systems. It frames the VideoLLM memory-eviction problem as structurally identical to the shared-CPU-cache side-channel problem solved by Prime+Probe/Evict+Time-style timing attacks, and states that it carries over the two standard hardware defenses — partitioned caches (bounding any one user's/domain's share of a shared cache) and randomized/keyed caches (mapping addresses with a secret key so an attacker cannot predict which lines are evicted) — directly into the VideoLLM's token memory as KeyMem's two layers. The threat model section also explicitly invokes Kerckhoffs's principle (the adversary is assumed to know everything about the system except the secret key) to justify evaluating the keyed-sampling layer.

- source_quote: "Processor caches faced both questions when Prime+Probe and Evict+Time showed that an attacker who controls which lines enter a shared cache also controls which lines leave it. Hardware answered with partitioned caches that bound the share of any one domain and with randomized caches that map addresses with a secret key. We carry both ideas into the token memory of a VideoLLM as two separate layers and call the result KeyMem."
- source_quote_2: "Following Kerckhoffs's principle, the adversary knows the model, the base policy, the budget, the quota, the bucket grid and every hyperparameter, and it chooses both the content of its segment and the moment at which the segment starts."
- source_url: https://arxiv.org/pdf/2611.25649
- source_sha: 3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f
- source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
- source_author: Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang
- source_date: 2026-11-19
- source_venue: "arXiv:2611.25649v1 [cs.CY]"
- source_tier: 1
- Note: this claim describes KeyMem's own stated lineage/design intent (directly quoted, Tier 1 for that framing claim). It does not independently verify what Osvik, Shamir & Tromer's 2006 "Cache attacks and countermeasures: the case of AES" (the Prime+Probe/Evict+Time source the paper cites as ref. [46]) or Kiriansky et al.'s DAWG (partitioned caches, ref. [33]) or Qureshi's CEASER (randomized keyed caches, ref. [49]) actually showed — those are the foundational prior works this paper measures itself against, and none was independently fetched this session. See Entity candidates below.

## Claim: Secret-keyed sampling measurably helps only against a simulating adaptive adversary, not under natural stress, and even there does not reliably beat simple deterministic quotas

Under natural stress conditions (the five stressors above), the paper reports that the keyed draw adds no measurable benefit over deterministic temporal quotas — "the 18 paired intervals of KeyMem against the two deterministic quotas all include zero." The keyed sampling only shows a measurable advantage under a specifically constructed adaptive adversary that is assumed (per Kerckhoffs's principle) to know the model, policy, budget, quota and bucket grid, and that simulates the memory offline to choose which natural footage to insert and when — and even there, a secret key does not consistently outperform the best deterministic quota, only a leaked one.

- source_quote: "Deterministic and keyed quotas perform comparably under natural stress. Under an adaptive adversary, a secret key preserves 4.4–9.5% more early-clip accuracy than a leaked key, but does not consistently beat the best deterministic quota."
- source_quote_2: "the 18 paired intervals of KeyMem against the two deterministic quotas all include zero"
- source_url: https://arxiv.org/pdf/2611.25649
- source_sha: 3f9e83f2dc2afdc30540eea3864c201947827ba785f356504d34094129ddfa2f
- source_title: "KeyMem: Temporal Occupancy Control for Robust Streaming VideoLLM Memory"
- source_author: Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang
- source_date: 2026-11-19
- source_venue: "arXiv:2611.25649v1 [cs.CY]"
- source_tier: 1

## Further leads

- InfiniPot-V (Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang; NeurIPS 2026) — the strongest published retention baseline KeyMem compares against; compresses VideoLLM KV cache via temporal redundancy and value norms. Not independently fetched — per KeyMem, ref. [32].
- Nexus Sampling (Duc Duong, Hoang Anh Duy Le, Jianwen Xie, Anshumali Shrivastava, Zhaozhuo Xu; EMNLP 2026, "Forget without compromise") — weighted reservoir sampling for KV-cache eviction under fixed budgets; the weaker published baseline in KeyMem's tests, drawing from a public seed (no secrecy). Per KeyMem, ref. [21].
- DeltaS (Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim, Moon Hwan Kim; arXiv:2609.27470, 2026) — the "soft quota over elapsed time" (equal-spans allocation without an upper bound) that KeyMem's hard cap is explicitly contrasted against. Per KeyMem, ref. [35].
- Dobre, Schwinn, Gidel, Gella, Taslakian & Noël, "Visual memory attacks can persist through the KV cache" (arXiv:2610.09027, 2026) — an adjacent adversarial-ML finding cited in Related Work, that an adversarial image can act through the KV cache after it has left the context window.
- StreamingBench (Junming Lin et al.; ICASSP 2026) — the real-world streaming-QA benchmark (721 questions over 150 videos) KeyMem evaluates on outside its synthetic UCF101-based coverage benchmark; not independently fetched.
- The Gumbel-top-k sampling trick (Wouter Kool, Herke van Hoof, Max Welling; ICML 2019) underlying KeyMem's keyed-sampling layer, and its equivalence to Plackett-Luce weighted sampling-without-replacement (R. Duncan Luce 1959/2005; Robin L. Plackett 1975) — a 1970s statistics result repurposed here for cache-eviction security; not independently fetched.

## Entity candidates

- Dag Arne Osvik, Adi Shamir, Eran Tromer — people — authors of "Cache attacks and countermeasures: the case of AES" (RSA Conference Cryptographers' Track, 2006), the Prime+Probe/Evict+Time cache-timing-attack work KeyMem's entire two-layer design explicitly measures itself against and claims to carry forward into VideoLLM memory. Adi Shamir is also the "S" in RSA. This is the foundational prior-work figure this capture's hook actually rests on — flagged first, per the brief's instruction.
- Vladimir Kiriansky, Ilia Lebedev, Saman Amarasinghe, Srinivas Devadas, Joel Emer — people — authors of DAWG, "A defense against cache timing attacks in speculative execution processors" (2018), the partitioned-cache defense KeyMem's hard temporal cap is modeled on.
- Moinuddin K. Qureshi — person — author of CEASER, "Mitigating conflict-based cache attacks via encrypted-address and remapping," the randomized/keyed-cache defense KeyMem's keyed-sampling layer is modeled on.
- Auguste Kerckhoffs — person/concept — 19th-century cryptographer whose eponymous principle (a cryptosystem should remain secure even if everything about it except the key is public) is explicitly invoked by name to define this paper's adversary model.
- Wei Song, Yuxin Cao, Xiao Cheng, Christy Jie Liang, Xi Zheng, Leo Yu Zhang — people — this paper's own authors (Griffith University, National University of Singapore, Macquarie University, University of Technology Sydney).
- Minsoo Kim, Kyuhong Shim, Jungwook Choi, Simyung Chang — people — authors of InfiniPot-V, the strongest published baseline this paper benchmarks against.
- Wouter Kool, Herke van Hoof, Max Welling — people — originators of the Gumbel-top-k sampling trick that implements KeyMem's keyed layer.

No genuinely close collision was found with the retrieved related vault notes (the AI-supply-chain, hexels, Altair/BASIC, and rationality-continuity-problem notes surfaced by retrieval); this paper's subject — streaming-video-LLM KV-cache eviction and its borrowing from CPU cache-side-channel security — does not overlap their content, so no wikilinks were forced.
