Repeated query failures on a topic are a behavioral signature of a knowledge gap

gap-detectionknowledge-managementPKMepistemicsmetacognitionbehavioral-signals

When an information-seeking system repeatedly receives queries about a topic it cannot resolve well, that clustering pattern is a behavioral signal of absent knowledge — not a property of any single query, but a diagnostic emergent from the query log. Medhat et al. (Georgia Institute of Technology) demonstrate this by mapping conversational AI teaching-assistant questions onto a curriculum prerequisite graph and finding that "conversational AI interaction logs, mapped onto curriculum structure, carry actionable signals about topic-level knowledge gaps." The key insight is that the signal is behavioral rather than per-question: "a student asking many questions about a topic, or asking about it weeks after it was taught, is exhibiting a behavioral signature of difficulty."

Quantitative validation: topic-level question volume correlated with self-reported difficulty at Spearman's ρ = 0.491, p = 0.008, over n = 28 matched topics; the question stream was 1,340 question events from 164 students, and the difficulty measure an independent mid-semester survey of 280 respondents. (Precision added 2026-09-11 audit; the promotion wording ran the two figures together — "question frequency correlated with student self-reported difficulty at ρ = 0.491, p = 0.008, across 1,340 questions from 164 students".)

Scope of the source, for honesty about the title: Medhat et al. measure question volume per topic — help-seeking — and validate it against self-reported difficulty. They do not measure whether the assistant resolved the questions. The "queries it cannot resolve well" framing above, and the "query failures" of the title, are this note's extension to an information-seeking agent auditing its own gaps, not the paper's finding. (Added 2026-09-11 audit.)

The mechanism generalizes beyond tutoring. Any information-seeking agent that logs its queries can analyze that log for clustering. Clusters mark where the agent needed information it didn't have — which is, by definition, a gap. The signal is retrospective: the log must exist before it can be read. This distinguishes it from the structural method in claim-obligatory-attributes-as-gap-signal, which can be run at any time over existing notes without any behavioral trace.

Applied to SeekVault: each research batch run is a query session. When the batch agent searches the vault and returns no matching notes — or when the same topic recurs across multiple Further-leads sections without ever being promoted — those recurrences are behavioral signatures of gaps. A systematic log of unresolved vault searches would constitute the "interaction log" that Medhat et al. show to be gap-diagnostic. Currently SeekVault has no such log; implementing one would make this mechanism operational.

Together with claim-obligatory-attributes-as-gap-signal and claim-llm-explicit-implicit-gap-detection, this method covers a distinct failure mode: entire topics that never made it to notes at all, rather than gaps within existing notes. See question-gap-detection for the detection taxonomy these mechanisms collectively address.