GRADE detects LLM knowledge gaps by comparing gradient-subspace rank against hidden-state-subspace rank across layers

gap-detectiongradientsLLM-internalsrank-ratiobenchmark

GRADE's diagnostic: what a model has activated (hidden states) may not match what a query requires — and the gap is measurable as "the proportion of effective required updates against [the] activated knowledge." Per MLP layer, the gradient g = ∂L/∂W is projected into the hidden-state subspace; stable ranks of gradient and hidden state are compared as a per-layer rank ratio (srank(g)/srank(h)), and the cross-layer ratio vector feeds a small supervised gap detector. Across six benchmarks the method outperforms verbalized-confidence, probabilistic, and hidden-state baselines, with the largest gains on math reasoning, and is more robust to prompt paraphrase than hidden-state probes. (2026-09-11 audit: the paper's prose says only that "the effectiveness of our two methods aligns with task complexity" and that GRADEpos is the variant that pays off on "difficult reasoning datasets (MMLU, GSM8K and MATH)" (§4.2); "largest gains on math reasoning" is the vault's reading of Table 1's margins, not the authors' sentence. The paraphrase finding is the authors' own: "IC and Align-P exhibit notable performance fluctuations when input phrasing changes" (§4.2). Six datasets: GSM8K, MATH, MMLU, NQ, TQA, HotpotQA (§4.1).)

Two placement notes. First, this is the internalist branch of the gap-detection taxonomy — it reads the model's weights-versus-activations mismatch, where claim-llm-explicit-implicit-gap-detection reads corpora and claim-query-failure-clustering-as-gap-signal reads behavior; for a markdown vault it applies only "if SeekVault ever uses an embedded model as an index" (the promotion judgment of 2026-07-03 stands). Second, there is a pleasing symmetry the sources don't remark on: the gradient — training's own signal (claim-training-inference-compute-asymmetry-mechanism) — is here repurposed at inference time as a probe, a backward pass run not to learn but to ask what learning WOULD be required. Deployment cost of that backward pass is a queued open question. See moc-machine-self-knowledge.