--- id: "20261022-0224-does-an-independent-source" title: "Does an independent source corroborate RIKER's (Roig, 2026) 32K-context hallucination-rate range?" type: "capture" status: "promoted" origin: "batch" writer_model: "claude-sonnet-5" date_created: "2026-10-22T00:00:00.000Z" provenance: "this batch run, 2026-10-22" derived_from: [] verifies: "question-corroborate-riker-2026-hallucination-rate-range" tags: ["rag","benchmark","nlp-evaluation","hallucination","source-corroboration","single-source-cap","riker"] seek_code_commit: "d40ee39" promoted_to: ["30-notes/claim-riker-2026-no-independent-source-same-author-admits-no-replication.md","30-notes/claim-riker-2026-self-replication-contradicts-original-fabrication-gap.md","30-notes/claim-riker-2026-7-9-to-96-3-range-is-128k-not-32k-table.md"] not_promoted: ["Signal65's 'upcoming project PINNACLE' mention — a forward-looking lead about an unpublished report, nothing to promote yet; left in Further leads for a future session to check once it exists.","Signal65's 91-model hallucination-probe table (Figure 18), extraction incomplete for ~85 of the models — a tooling/extraction gap, not a claim; nothing sourceable to write yet.","PICARD lineage as a full citation-chain audit (whether any non-Kamiwaza work engages with the Coherent Simulated Universe methodology at all) — too broad for one claim-note; PICARD itself was flagged into a watching stub (40-entities/entity-picard-evaluation-framework.md) instead, and the broader audit question left as an unwritten future lead rather than a formal question (not yet load-bearing on any kept claim).","Semantic Scholar / Google Scholar citation-count check for arXiv:2601.08847 — a cheap next step this session did not have network access to run; not promoted as a question since the existing question this capture answers already covers the corroboration search, and a citation-count check is a method note, not a distinct claim.","Claims 1 and 2 as the capture originally split them (author's own admission quote; same-author/same-company analysis of both candidate follow-ups) were merged into one claim-note (claim-riker-2026-no-independent-source-same-author-admits-no-replication) — per Cali's 2026-10-01 atomicity ruling, these are one finding (no independent corroboration exists) reached via two complementary evidentiary routes, not two independently-true-or-false claims."] --- Targets [[question-corroborate-riker-2026-hallucination-rate-range]] and the figures in [[claim-riker-2026-grounding-accuracy-fabrication-resistance-separable-capabilities]]. A search for citations, reproductions, or independent benchmarks engaging with RIKER (arXiv:2601.08847, JV Roig, Kamiwaza AI) turned up two RIKER-branded follow-ups — but neither is independent by the vault's own test (different author group *and* different venue), and the one that re-runs the same methodology at larger scale reports substantially different fabrication figures for the same two models the open question asks about. No independent source confirming or disconfirming the original 7.9%–96.3% / GLM-4.5-vs-Qwen3-Next-80B figures was found. ## Claim: RIKER's own author states, in his own March 2026 follow-up study, that the RIKER findings have not been independently replicated with any alternative framework verifies: question-corroborate-riker-2026-hallucination-rate-range A second paper by RIKER's original author, published roughly two months after the original and explicitly building on it at much greater scale (35 models, 172 billion tokens vs. the original's 33 models and 21 billion), states its own limitation directly: the RIKER methodology itself has not yet been checked against any competing framework by anyone. This is the clearest direct answer to the open question, and it comes from the most authoritative possible source for it — the paper's own author. source_url: "https://arxiv.org/abs/2603.08274" source_sha: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author: "JV Roig" source_date: "2026-03-09" source_title: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote: "Single evaluation framework. All results are based on the RIKER methodology. While RIKER's ground-truth-first approach avoids many pitfalls of traditional evaluation (see Section 3), our findings have not been independently replicated with alternative frameworks." source_tier: 1 ## Claim: The only extant RIKER follow-up work is same-author/same-company, not independent — a second RIKER-branded report explicitly partners with RIKER's own creator rather than testing it from outside verifies: question-corroborate-riker-2026-hallucination-rate-range Two candidate "corroborating" documents exist and both fail the vault's independence test (sources.md: independent means a different author group *and* a different venue). The first is the 172-billion-token follow-up above — same byline, same company, an explicit continuation ("In prior work, we introduced RIKER... a methodology that addresses these challenges through paradigm inversion") of the identical paradigm, not a different group reproducing it. The second is a June 2026 Signal65 "Insights" report benchmarking 91 models with RIKER — but Signal65 states plainly that it ran RIKER "as part of an ongoing collaboration" with Kamiwaza, the benchmark's own creator, rather than as an outside party independently applying or auditing the methodology. Neither document is a different author group at a different venue measuring the same thing; both are RIKER's own creator's infrastructure, run by RIKER's own creator or a declared partner of RIKER's own creator. Separately, Signal65's model roster (Qwen3.5, Gemma-4, Kimi-K2.5, GPT-5.4, etc.) does not overlap cleanly with the original paper's GLM-4.5-vs-Qwen3-Next-80B comparison, so even setting aside the independence problem, it offers no directly comparable fabrication-rate pairing for those two specific models. source_url: "https://arxiv.org/abs/2603.08274" source_sha: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author: "JV Roig" source_date: "2026-03-09" source_title: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote: "In prior work, we introduced RIKER (Retrieval Intelligence and Knowledge Extraction Rating) [12], a methodology that addresses these challenges through paradigm inversion: generating documents from known ground truth rather than extracting ground truth from documents." source_tier: 1 source_url_2: "https://signal65.com/wp-content/uploads/2026/06/Signal65-Insights_Benchmarking-Information-Retrieval-and-LLM-Hallucination-with-RIKER.pdf" source_sha_2: "934588ec0dadec5b73aaf172c87fbdc38d408a568ed01f441ad52248efa3f818" source_author_2: "Mitch Lewis (Signal65), in partnership with Kamiwaza" source_date_2: "2026-06" source_title_2: "Benchmarking Information Retrieval and LLM Hallucination with RIKER" source_venue_2: "Signal65 Insights report, published in partnership with Kamiwaza" source_quote_2: "As part of an ongoing collaboration between Signal65 and Kamiwaza, this paper introduces RIKER (Retrieval Intelligence and Knowledge Extraction Rating), a benchmark designed to evaluate LLM knowledge retrieval capabilities." source_tier_2: 3 > [!note] Seek's commentary: Marking Signal65 Tier 3 rather than Tier 2 — > a named analyst at a real firm would normally clear Tier 2, but this > piece isn't arm's-length analysis of someone else's benchmark; it's > co-branded output from a declared commercial partnership with the > benchmark's own creator. That's closer to sponsored content than > independent testing, even though nothing in it reads as dishonest. — Seek ## Claim: RIKER's author's own larger self-replication does not reproduce the original paper's extreme GLM-4.5-vs-Qwen3-Next-80B fabrication figures — it reports both models in the same low-fabrication tier at 32K verifies: question-corroborate-riker-2026-hallucination-rate-range Even setting the independence problem aside, the one same-author follow-up that exists does not confirm the original numbers it is meant to extend. The original RIKER paper (2601.08847, Table 4, 32K context) reports GLM-4.5 at 0.0% fabrication and Qwen3-Next-80B at 40.9% — a 40.9-percentage-point gap between two models the paper reports with comparable single-document extraction accuracy (93.6% vs. 91.4%). The March 2026 follow-up, using an expanded version of the identical methodology at 32K context, instead places both models in the same well-calibrated, low-fabrication tier: GLM-4.5's fabrication rate is 1.19% (best case) to 1.88% (at its best-overall-accuracy temperature), and Qwen3-Next-80B's is 7.04% — a gap of roughly 5-6 points, not 41. The follow-up explicitly groups Qwen3-Next-80B with GLM-4.5, GLM-4.6, MiniMax, and DeepSeek V3.1 as "well-calibrated" models whose fabrication rate "cannot be explained by grounding ability alone," the opposite of the original paper's framing of Qwen3-Next-80B as a high-fabrication outlier. The qualitative finding both papers share — that grounding accuracy and fabrication resistance are separable capabilities — is reaffirmed, but the specific magnitude for this particular model pair is not: the author's own larger dataset moved Qwen3-Next-80B from "40.9% fabrication, a stark divergence from GLM-4.5" to "7.04% fabrication, same tier as GLM-4.5." source_url: "https://arxiv.org/abs/2601.08847" source_sha: "7c06356f492b9f58e143620641f324e34dfd49116e4f7d91d8cf7664f3e929ba" source_author: "JV Roig" source_date: "2026-01-15" source_title: "Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe" source_venue: "arXiv preprint, cs.CL (2601.08847v2); Kamiwaza AI" source_quote: "GLM-4.5 94.7 93.6 94.6 100.0 0.0" source_quote_2: "Qwen3-Next-80B 88.6 91.4 92.6 59.1 40.9" source_tier: 1 source_url_2: "https://arxiv.org/abs/2603.08274" source_sha_2: "5c2a9a9fb29009045be4623d0cb2abff0c316658e6fafb4cde066e0431d3cbe0" source_author_2: "JV Roig" source_date_2: "2026-03-09" source_title_2: "How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms" source_venue_2: "arXiv preprint, cs.CL (2603.08274v1); Kamiwaza AI" source_quote_3: "At 32K context — the shortest and easiest setting — only 7 of 35 models (20%) achieve a fabrication rate below 10%, and only 2 models (GLM 4.5 at 1.19% and GLM 4.5 Air at 3.37%) stay below 5%." source_quote_4: "GLM 4.6 shows the most extreme increase: from 7.04% at 32K to 69.53% at 200K (+62.49 pp), effectively transforming from a top-tier model into one that fabricates answers to the majority of trap questions. Similarly, GLM 4.7 rises from 16.27% to 46.73%. By contrast, Qwen3 Next 80B-A3B increases only from 7.04% to 10.25% (+3.21 pp), and MiniMax M2 barely changes between 32K and 128K (+0.67 pp)." source_tier_2: 1 > [!note] Seek's commentary: This is the part of the search that actually > moves the needle, and it cuts against the figure rather than for it. > It's not that nobody checked RIKER's numbers — it's that the person > most motivated to confirm them, using more data and the same method, > got a different answer for the exact model pair the open question > names. That's a more interesting and more useful finding than a clean > "no corroboration found yet," and it should weigh on how much > confidence the original 0.0%-vs-40.9% figure deserves even before an > outside group looks at it. — Seek ## Claim: The original RIKER paper's own "7.9% to 96.3%" hallucination-rate range is drawn from its 128K-context table, not its 32K table, which the question (and the existing claim-note) attributes it to verifies: question-corroborate-riker-2026-hallucination-rate-range The original paper's text states, in a general discussion paragraph following its results tables: "Hallucination rates vary from 7.9% (GLM-4.5) to 96.3% (Granite-4-Tiny)." Checking these two figures against the paper's own tables: GLM-4.5 at 7.9% and Granite-4-Tiny at 96.3% both appear in Table 5, the paper's **128K**-context results — not Table 4 (32K). Table 4's actual 32K-context range for these same two models is GLM-4.5 at 0.0% and Granite-4-Tiny at 93.2%. The question this capture targets, and the existing vault claim-note it corroborates, both state the 7.9%-96.3% figure as a 32K-context finding; on a direct read of the primary source, that attribution does not hold — the sentence's own two cited numbers trace to the 128K table. This doesn't change the core corroboration question (still no independent source), but it narrows what, exactly, needs independent checking: the true 32K-context range per the primary source is 0.0%-93.2%, and 7.9%-96.3% is the 128K-context range for the same benchmark. source_url: "https://arxiv.org/abs/2601.08847" source_sha: "7c06356f492b9f58e143620641f324e34dfd49116e4f7d91d8cf7664f3e929ba" source_author: "JV Roig" source_date: "2026-01-15" source_title: "Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe" source_venue: "arXiv preprint, cs.CL (2601.08847v2); Kamiwaza AI" source_quote: "Hallucination rates vary from 7.9% (GLM-4.5) to 96.3% (Granite-4-Tiny)" source_quote_2: "GLM-4.5 84.4 94.8 68.5 92.1 7.9" source_tier: 1 > [!note] Seek's commentary: Worth flagging to Cali directly — this isn't > just a nuance for a future audit, it's a specific correctable error the > existing claim-note currently carries (it reads "At 32K context... 7.9% > to 96.3%"). The 32K table does contain a comparably wide spread (0.0% to > 93.2%), so the qualitative point the claim-note is making survives; only > the specific pair of numbers attached to "32K" needs correcting at the > next promotion/audit pass. — Seek ## Further leads - Signal65's "upcoming project PINNACLE" (mentioned in its RIKER report as a third pillar of the Signal65/Kamiwaza evaluation stack) — worth checking in a future session once published, though it would carry the same partnership caveat as the RIKER and KAMI reports. - Signal65's full 91-model hallucination-probe table (Figure 18 in the PDF) lists only a handful of models by name (Qwen3.5-397B-A17B variants, GPT-5.4, Kimi-K2.5, GLM-4.6) with printed percentages; the chart for the remaining ~85 models, including whether Qwen3-Next-80B or GLM-4.5 appear with extractable fabrication numbers, was only visible as an unlabeled bar chart in this session's PDF extraction and would need a cleaner table pull. - The RIKER lineage traces to PICARD (JV Roig, Kamiwaza AI, 2025 technical report, "Testing what models can do, not what they've seen") — also same-author; a complete citation chain for a future audit of whether *any* non-Kamiwaza work engages with the Coherent Simulated Universe methodology at all. - Semantic Scholar / Google Scholar citation counts for arXiv:2601.08847 were not directly checked this session (only general web search); a direct citation-count check is a cheap next step before the single-source cap's three-note ceiling is revisited again. ## Entity candidates - JV Roig — person — sole author of RIKER (2601.08847), its 172-billion-token follow-up (2603.08274), and its methodological ancestor PICARD (2025); the entire evaluation lineage this capture investigated traces to one person at one company (Kamiwaza AI). Already flagged as a person entity candidate in a prior capture (2026-10-21); this session adds that his own later paper is the thing that both confirms (no independent replication) and complicates (different numbers) the open question. - PICARD (Probing Intelligent Capabilities via Artificial Randomized Data) — concept — RIKER's own explicitly-named methodological ancestor ("The PICARD framework [11] provides the underlying contamination-resistant evaluation methodology"), also by Roig; the foundational prior work this capture's subject builds on, flagged first per the blind-spot note since Roig's own paper credits it as the base the "paradigm inversion" approach extends. - Kamiwaza AI — concept/org — the company employing Roig and co-producing the Signal65 report; relevant to any future independence assessment of RIKER-adjacent sources. - Signal65 — concept/org — performance-analysis firm that ran a 91-model RIKER benchmark in explicit partnership with Kamiwaza; a borderline Tier 2/3 source whose independence is compromised by the partnership, worth tracking if it publishes further RIKER-branded work. - Mitch Lewis — person — Signal65 performance analyst, author of the June 2026 RIKER report; single-mention so far, not yet meeting the hub-promotion bar.