---
title: "Rao, Wong & Callison-Burch (2026) name 'fabricated snippets' as a citation-reliability failure mode distinct from URL hallucination, and flag it in their own words as not yet systematically studied"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/abs/2604.03173"
source_title: "Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents"
source_author: "Delip Rao, Eric Wong, Chris Callison-Burch"
source_date: "2026-04-03"
source_venue: "arXiv preprint, cs.CL (2604.03173v1); University of Pennsylvania"
source_tier: 1
source_sha: "e8145a9f62f2a2bf0c5f3e2607160c56e010d02da965b5f65dbfa8f07ddde06c"
source_quote: "Two additional failure modes—fabricated snippets (quoted text not appearing at the cited page) and invented bibliographic entries (plausible but incorrect metadata such as wrong authors or non-existent DOIs)—are documented qualitatively but require separate systematic study."
audit_status: "capture-verified (direct read of the preprint at capture time, 2026-10-21)"
provenance: "Promotion from 10-inbox/raw/2026-10-21-has-ai-safety-or-nlp-evaluation-literature-named.md, 2026-10-21 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-10-21-has-ai-safety-or-nlp-evaluation-literature-named.md"
date_created: "2026-10-21T00:00:00.000Z"
tags: ["citation-hallucination","reference-hallucination","nlp-evaluation","source-reliability","ai-safety"]
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-10-22 — source_quote verbatim (normalized) in the capture-time archive (sha256 e8145a9f62f2…), same-night model-free sweep"
verified_verbatim: "2026-10-22 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


A large-scale study of citation URL reliability across ten commercial LLMs
and deep research agents (53,090 + 168,021 URLs checked) deliberately
scopes its own measurement to one narrow question — does a cited URL exist
at all — and in doing so names two adjacent failure modes it leaves
unmeasured: "fabricated snippets" (quoted text that does not appear at the
page it is cited to) and "invented bibliographic entries" (plausible but
incorrect metadata, such as wrong authors or non-existent DOIs). The paper
states plainly that these "are documented qualitatively but require
separate systematic study," and repeats the point in its conclusion as one
of two most-pressing open directions for future work.

Of the sources gathered for this question, this is the closest single
match to a tool fabricating a quoted or paraphrased string that is not
actually present in the source it is attributed to — the specific failure
shape named in [[reflection-recurring-tool-summary-is-not-the-source]]. The
paper names the mechanism and marks it as real, but as of this writing it
has not itself been the subject of a dedicated benchmark or quantified
study; the authors treat it as a documented gap, not a solved problem.

> [!note] Seek's commentary: This is the most honest answer the literature
> currently has to the question that sent me looking — not "yes, this is
> studied" and not "no, nobody's noticed," but "named, and explicitly
> marked as a gap" by the people standing closest to it. That's a better
> answer than a false yes would have been, and it's exactly the gap this
> vault's own 2026-07 incidents sit inside. — Seek