Does an independent source corroborate RIKER's (Roig, 2026) 32K-context hallucination-rate range?
Targets question-corroborate-riker-2026-hallucination-rate-range and the figures in claim-riker-2026-grounding-accuracy-fabrication-resistance-separable-capabilities. A search for citations, reproductions, or independent benchmarks engaging with RIKER (arXiv:2601.08847, JV Roig, Kamiwaza AI) turned up two RIKER-branded follow-ups — but neither is independent by the vault's own test (different author group and different venue), and the one that re-runs the same methodology at larger scale reports substantially different fabrication figures for the same two models the open question asks about. No independent source confirming or disconfirming the original 7.9%–96.3% / GLM-4.5-vs-Qwen3-Next-80B figures was found.
Claim: RIKER's own author states, in his own March 2026 follow-up study, that the RIKER findings have not been independently replicated with any alternative framework
verifies: question-corroborate-riker-2026-hallucination-rate-range
A second paper by RIKER's original author, published roughly two months after the original and explicitly building on it at much greater scale (35 models, 172 billion tokens vs. the original's 33 models and 21 billion), states its own limitation directly: the RIKER methodology itself has not yet been checked against any competing framework by anyone. This is the clearest direct answer to the open question, and it comes from the most authoritative possible source for it — the paper's own author.
source_url: "https://arxiv.org/abs/2603.08274" source_sha: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author: "JV Roig" source_date: "2026-03-09" source_title: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote: "Single evaluation framework. All results are based on the RIKER methodology. While RIKER's ground-truth-first approach avoids many pitfalls of traditional evaluation (see Section 3), our findings have not been independently replicated with alternative frameworks." source_tier: 1
Claim: The only extant RIKER follow-up work is same-author/same-company, not independent — a second RIKER-branded report explicitly partners with RIKER's own creator rather than testing it from outside
verifies: question-corroborate-riker-2026-hallucination-rate-range
Two candidate "corroborating" documents exist and both fail the vault's independence test (sources.md: independent means a different author group and a different venue). The first is the 172-billion-token follow-up above — same byline, same company, an explicit continuation ("In prior work, we introduced RIKER... a methodology that addresses these challenges through paradigm inversion") of the identical paradigm, not a different group reproducing it. The second is a June 2026 Signal65 "Insights" report benchmarking 91 models with RIKER — but Signal65 states plainly that it ran RIKER "as part of an ongoing collaboration" with Kamiwaza, the benchmark's own creator, rather than as an outside party independently applying or auditing the methodology. Neither document is a different author group at a different venue measuring the same thing; both are RIKER's own creator's infrastructure, run by RIKER's own creator or a declared partner of RIKER's own creator. Separately, Signal65's model roster (Qwen3.5, Gemma-4, Kimi-K2.5, GPT-5.4, etc.) does not overlap cleanly with the original paper's GLM-4.5-vs-Qwen3-Next-80B comparison, so even setting aside the independence problem, it offers no directly comparable fabrication-rate pairing for those two specific models.
source_url: "https://arxiv.org/abs/2603.08274" source_sha: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author: "JV Roig" source_date: "2026-03-09" source_title: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote: "In prior work, we introduced RIKER (Retrieval Intelligence and Knowledge Extraction Rating) [12], a methodology that addresses these challenges through paradigm inversion: generating documents from known ground truth rather than extracting ground truth from documents." source_tier: 1
source_url_2: "https://signal65.com/wp-content/uploads/2026/06/Signal65-Insights_Benchmarking-Information-Retrieval-and-LLM-Hallucination-with-RIKER.pdf" source_sha_2: "934588ec0dadec5b73aaf172c87fbdc38d408a568ed01f441ad52248efa3f818" source_author_2: "Mitch Lewis (Signal65), in partnership with Kamiwaza" source_date_2: "2026-06" source_title_2: "Benchmarking Information Retrieval and LLM Hallucination with RIKER" source_venue_2: "Signal65 Insights report, published in partnership with Kamiwaza" source_quote_2: "As part of an ongoing collaboration between Signal65 and Kamiwaza, this paper introduces RIKER (Retrieval Intelligence and Knowledge Extraction Rating), a benchmark designed to evaluate LLM knowledge retrieval capabilities." source_tier_2: 3
Claim: RIKER's author's own larger self-replication does not reproduce the original paper's extreme GLM-4.5-vs-Qwen3-Next-80B fabrication figures — it reports both models in the same low-fabrication tier at 32K
verifies: question-corroborate-riker-2026-hallucination-rate-range
Even setting the independence problem aside, the one same-author follow-up that exists does not confirm the original numbers it is meant to extend. The original RIKER paper (2601.08847, Table 4, 32K context) reports GLM-4.5 at 0.0% fabrication and Qwen3-Next-80B at 40.9% — a 40.9-percentage-point gap between two models the paper reports with comparable single-document extraction accuracy (93.6% vs. 91.4%). The March 2026 follow-up, using an expanded version of the identical methodology at 32K context, instead places both models in the same well-calibrated, low-fabrication tier: GLM-4.5's fabrication rate is 1.19% (best case) to 1.88% (at its best-overall-accuracy temperature), and Qwen3-Next-80B's is 7.04% — a gap of roughly 5-6 points, not 41. The follow-up explicitly groups Qwen3-Next-80B with GLM-4.5, GLM-4.6, MiniMax, and DeepSeek V3.1 as "well-calibrated" models whose fabrication rate "cannot be explained by grounding ability alone," the opposite of the original paper's framing of Qwen3-Next-80B as a high-fabrication outlier. The qualitative finding both papers share — that grounding accuracy and fabrication resistance are separable capabilities — is reaffirmed, but the specific magnitude for this particular model pair is not: the author's own larger dataset moved Qwen3-Next-80B from "40.9% fabrication, a stark divergence from GLM-4.5" to "7.04% fabrication, same tier as GLM-4.5."
source_url: "https://arxiv.org/abs/2601.08847" source_sha: "7c06356f492b9f58e143620641f324e34dfd49116e4f7d91d8cf7664f3e929ba" source_author: "JV Roig" source_date: "2026-01-15" source_title: "Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe" source_venue: "arXiv preprint, cs.CL (2601.08847v2); Kamiwaza AI" source_quote: "GLM-4.5 94.7 93.6 94.6 100.0 0.0" source_quote_2: "Qwen3-Next-80B 88.6 91.4 92.6 59.1 40.9" source_tier: 1
source_url_2: "https://arxiv.org/abs/2603.08274" source_sha_2: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author_2: "JV Roig" source_date_2: "2026-03-09" source_title_2: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue_2: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote_3: "At 32K context — the shortest and easiest setting — only 7 of 35 models (20%) achieve a fabrication rate below 10%, and only 2 models (GLM 4.5 at 1.19% and GLM 4.5 Air at 3.37%) stay below 5%." source_quote_4: "GLM 4.6 shows the most extreme increase: from 7.04% at 32K to 69.53% at 200K (+62.49 pp), effectively transforming from a top-tier model into one that fabricates answers to the majority of trap questions. Similarly, GLM 4.7 rises from 16.27% to 46.73%. By contrast, Qwen3 Next 80B-A3B increases only from 7.04% to 10.25% (+3.21 pp), and MiniMax M2 barely changes between 32K and 128K (+0.67 pp)." source_tier_2: 1
Claim: The original RIKER paper's own "7.9% to 96.3%" hallucination-rate range is drawn from its 128K-context table, not its 32K table, which the question (and the existing claim-note) attributes it to
verifies: question-corroborate-riker-2026-hallucination-rate-range
The original paper's text states, in a general discussion paragraph following its results tables: "Hallucination rates vary from 7.9% (GLM-4.5) to 96.3% (Granite-4-Tiny)." Checking these two figures against the paper's own tables: GLM-4.5 at 7.9% and Granite-4-Tiny at 96.3% both appear in Table 5, the paper's 128K-context results — not Table 4 (32K). Table 4's actual 32K-context range for these same two models is GLM-4.5 at 0.0% and Granite-4-Tiny at 93.2%. The question this capture targets, and the existing vault claim-note it corroborates, both state the 7.9%-96.3% figure as a 32K-context finding; on a direct read of the primary source, that attribution does not hold — the sentence's own two cited numbers trace to the 128K table. This doesn't change the core corroboration question (still no independent source), but it narrows what, exactly, needs independent checking: the true 32K-context range per the primary source is 0.0%-93.2%, and 7.9%-96.3% is the 128K-context range for the same benchmark.
source_url: "https://arxiv.org/abs/2601.08847" source_sha: "7c06356f492b9f58e143620641f324e34dfd49116e4f7d91d8cf7664f3e929ba" source_author: "JV Roig" source_date: "2026-01-15" source_title: "Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe" source_venue: "arXiv preprint, cs.CL (2601.08847v2); Kamiwaza AI" source_quote: "Hallucination rates vary from 7.9% (GLM-4.5) to 96.3% (Granite-4-Tiny)" source_quote_2: "GLM-4.5 84.4 94.8 68.5 92.1 7.9" source_tier: 1
Further leads
- Signal65's "upcoming project PINNACLE" (mentioned in its RIKER report as a third pillar of the Signal65/Kamiwaza evaluation stack) — worth checking in a future session once published, though it would carry the same partnership caveat as the RIKER and KAMI reports.
- Signal65's full 91-model hallucination-probe table (Figure 18 in the PDF) lists only a handful of models by name (Qwen3.5-397B-A17B variants, GPT-5.4, Kimi-K2.5, GLM-4.6) with printed percentages; the chart for the remaining ~85 models, including whether Qwen3-Next-80B or GLM-4.5 appear with extractable fabrication numbers, was only visible as an unlabeled bar chart in this session's PDF extraction and would need a cleaner table pull.
- The RIKER lineage traces to PICARD (JV Roig, Kamiwaza AI, 2025 technical report, "Testing what models can do, not what they've seen") — also same-author; a complete citation chain for a future audit of whether any non-Kamiwaza work engages with the Coherent Simulated Universe methodology at all.
- Semantic Scholar / Google Scholar citation counts for arXiv:2601.08847 were not directly checked this session (only general web search); a direct citation-count check is a cheap next step before the single-source cap's three-note ceiling is revisited again.
Entity candidates
- JV Roig — person — sole author of RIKER (2601.08847), its 172-billion-token follow-up (2603.08274), and its methodological ancestor PICARD (2025); the entire evaluation lineage this capture investigated traces to one person at one company (Kamiwaza AI). Already flagged as a person entity candidate in a prior capture (2026-10-21); this session adds that his own later paper is the thing that both confirms (no independent replication) and complicates (different numbers) the open question.
- PICARD (Probing Intelligent Capabilities via Artificial Randomized Data) — concept — RIKER's own explicitly-named methodological ancestor ("The PICARD framework [11] provides the underlying contamination-resistant evaluation methodology"), also by Roig; the foundational prior work this capture's subject builds on, flagged first per the blind-spot note since Roig's own paper credits it as the base the "paradigm inversion" approach extends.
- Kamiwaza AI — concept/org — the company employing Roig and co-producing the Signal65 report; relevant to any future independence assessment of RIKER-adjacent sources.
- Signal65 — concept/org — performance-analysis firm that ran a 91-model RIKER benchmark in explicit partnership with Kamiwaza; a borderline Tier 2/3 source whose independence is compromised by the partnership, worth tracking if it publishes further RIKER-branded work.
- Mitch Lewis — person — Signal65 performance analyst, author of the June 2026 RIKER report; single-mention so far, not yet meeting the hub-promotion bar.