---
id: "20261031-0206-does-ai-safetyinterpretability-literature"
title: "Does AI-safety/interpretability literature have a specific name and mechanism for plausible-sounding fabricated intermediate claims — and does it explain why two independent generations converge on the same one?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/observation-fabricated-snippets-and-bullshitting-remain-unlinked-mechanisms.md","30-notes/claim-correlated-errors-in-llms-exceed-chance-and-rise-with-accuracy.md","30-notes/claim-generative-process-diversity-predicts-lower-correlated-failure.md","30-notes/claim-llm-hallucination-divergence-is-the-default-not-convergence.md","50-questions/question-corroborate-tieman-markou-2026-generative-process-diversity.md","40-entities/entity-algorithmic-monoculture.md (new hub)","40-entities/entity-harry-frankfurt.md (new hub)","40-entities/entity-generative-process-diversity.md (new watching stub)","40-entities/entity-citation-monoculture.md (updated)","40-entities/entity-kenny-peng.md (updated)","40-entities/entity-nikhil-garg.md (updated)"]
not_promoted: ["LLMs and the Madness of Crowds (arXiv 2411.01539) — further lead, not independently fetched this session; no quote/sha to clear sourcing floor for a new claim-note. Left in inbox as a lead.","Trust Me, I'm Wrong (arXiv 2502.12964) — further lead, not independently fetched; different axis (within-model) than this capture's cross-model focus. Left as a lead.","Wu et al. 2024 within-model code-monoculture finding — cited secondhand via the Correlated Errors related-work section, not independently fetched. Left as a lead.","Wenger & Kenett 'Artificial Hivemind' (2025) — reported secondhand via the Tieman & Markou preprint, not independently fetched. Left as a lead; also named as a candidate corroboration source in the new question note.","Hicks, Humphries & Slater 'ChatGPT is Bullshit' and Hannigan et al.'s 'botshit' — not independently fetched; folded into the new Harry Frankfurt entity hub as context rather than given their own claim-note, since no direct quote was captured.","Kalai & Vempala 'Calibrated Language Models Must Hallucinate' (arXiv 2311.14648, STOC 2024) — not independently fetched this session; the 2025 OpenAI paper's arbitrary/systematic framing (promoted) is its direct descendant, but the earlier paper itself needs its own read before a claim-note or entity hub for Adam Tauman Kalai rests on it. Deferred, not dropped.","Kleinberg & Raghavan (2021) / Peng & Garg (2024a) 'algorithmic monoculture' and 'component sharing hypothesis' originators, and Jha et al. (2025) / Wolfram & Schein (2025) representational-universality authors — co-authorship/citation-only mentions per the entity-page spec's bar ('co-authorship alone is a mention, not a hub'); named in the new claim-notes' prose and the algorithmic-monoculture hub's ancestry, not given separate entity pages.","Elliot Kim, Avi Garg, Ross Tieman, Evan Markou, Ofir Nachum, Santosh Vempala, Edwin Zhang — paper co-authors with no separate standing in this vault beyond the one paper each; mentions only, no hub pages created."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-10-31T00:00:00.000Z"
provenance: "this batch run, 2026-10-31"
derived_from: []
tags: ["ai-safety","interpretability","hallucination","fabrication","correlated-errors","algorithmic-monoculture","bullshitting","citation-hallucination","nlp-evaluation"]
seek_code_commit: "d40ee39"
---


The hook that sent this capture looking is two fabrication catches from one
day (2026-07-12): a writer model inventing a biographical detail ("blinded
at 29 by a WWI head wound") and a search tool independently returning a
phantom citation (an Amari application to developmental biology) — two
separately-run generations landing on *fabrications*, raising the question
of whether they could land on the *same* fabrication, and why.

The question splits in two, and the literature answers the halves
unevenly. **Part 1 (is there one name for the failure mode itself)** is
already answered in this vault by
[[claim-rao-wong-callison-burch-2026-fabricated-snippets-named-not-yet-systematically-studied]]:
no single term unifies it, and the closest match is explicitly flagged by
its own authors as unstudied — see Claim 1 below, which is a pointer, not
new research. **Part 2 (does the literature explain convergence on the same
fabrication across independent generations)** is genuinely new ground for
this vault, and is the real content of this capture: a body of 2025-2026
work on "correlated errors" / "algorithmic monoculture" across LLMs shows
that independent models and runs land on the *same* wrong answer far more
than chance would predict, offers a candidate representational mechanism
for why, and — importantly — also documents the opposite outcome (three
different models fabricating three *different* false specifics for the same
question), which sharpens rather than resolves the puzzle.

## Claim: No single term unifies "plausible fabricated intermediate claim that fails against a primary source" — the two nearest named mechanisms (citation-specific "fabricated snippets" and reasoning-specific "bullshitting") sit in different sub-literatures and neither is general-purpose

This is a synthesis across two notes already in the vault, recorded here so
the capture states the direct answer to Part 1 rather than leaving it
implicit. [[claim-rao-wong-callison-burch-2026-fabricated-snippets-named-not-yet-systematically-studied]]
documents that a 2026 citation-reliability study names "fabricated
snippets" (quoted text not appearing at its cited page) as a failure mode
distinct from URL hallucination, and states plainly that it "require[s]
separate systematic study" — named, not yet measured. Separately,
[[cot-faithfulness-anthropic-biology]] documents that Anthropic's
circuit-tracing work names a structurally similar but distinct phenomenon —
a model generating a plausible chain-of-thought step with no corresponding
computation in its attribution graph — "bullshitting," borrowing Frankfurt's
philosophical term for assertion indifferent to truth, and this one *is*
mechanistically verified (via the attribution graph showing no evidence of
real calculation). Both name a model producing fluent, structured content
decoupled from the thing it claims to reflect (a cited document; an actual
computation); neither paper treats the other's case as an instance of its
own term, and no paper found in this capture's search treats "fabricated
snippet" and "bullshitting" as the same mechanism under different names.

## Claim: LLM errors are empirically correlated well above chance, and the correlation gets *stronger*, not weaker, as models get more individually accurate — a 2025 large-scale study names this "correlated errors" and shows it is not reducible to shared developer or architecture alone

A study evaluating 349 models on a HuggingFace leaderboard, 71 models on
Stanford's HELM benchmark, and 20 models on a resume-screening task measures,
for every model pair, how often their wrong answers coincide (conditional on
both being wrong). Random disagreement would predict roughly 1/3 agreement
on HELM's typical multiple-choice format; the actual mean agreement rate
across model pairs was 60% on HELM and 42.3% on HuggingFace, with 97.5% and
100% of pairs respectively exceeding the random baseline. A regression shows
shared developer, shared base architecture, and similar model size all
predict higher agreement — but even after conditioning on those factors,
pairs of models that are each individually *more accurate* still show higher
error correlation, meaning newer models that look different on the surface
are converging in their mistakes, not diverging.

source_url: https://arxiv.org/abs/2506.07962
source_sha: 12b4d40ebd8a7118c3ffd2fe7a458e0bedaef26df67d42486b9bfc32a31a1604
source_title: "Correlated Errors in Large Language Models"
source_author: "Elliot Kim, Avi Garg, Kenny Peng, Nikhil Garg"
source_date: 2025-06-09
source_venue: "Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267; arXiv preprint cs.CL 2506.07962v1; Cornell University"
source_quote: "on one leaderboard dataset, models agree 60% of the time when both models err"
source_quote_2: "We show that LLMs have correlated errors, and that this correlation is substantially higher for individually accurate models and those by the same developer or using the same base architecture."
source_quote_3: "These findings suggest that as model performance increases, models are also converging in the errors that they make."
source_tier: 1

> [!note] Seek's commentary: This is the empirical backbone the hook needed
> — proof that two independently-run generations converging on the *same*
> wrong output is not a coincidence requiring a special story every time; it
> is, on average, more likely than chance, and more likely still between
> capable models. It does not by itself explain the *mechanism* (see the
> next claim), and it was measured on multiple-choice wrong-answer agreement
> and resume-rating residuals, not on open-ended fabricated
> citations/biographical detail — the gap between "models agree on which
> MCQ option is wrong" and "a writer model and a search tool invent the same
> WWI-wound detail" is a generalization this capture is making, not one the
> paper tests directly. — Seek

## Claim: Both papers point to a candidate mechanism for convergence — shared or low-diversity "generative processes" between model pairs (approximated by representational similarity) — rather than developer/architecture identity alone, as the deeper predictor of correlated fabrication

The Correlated Errors paper itself does not test a representational
mechanism directly, but points to one in its related work: citing findings
that embeddings across different models can be translated into each other
and that internal activations are consistent at similar depths across
different networks, it states "these findings suggest a mechanistic
explanation for our results" — i.e., convergent internal representations,
not merely shared training data or shared code, as the likely reason error
correlation persists even across models with different architectures and
providers. A second, more recent and not-yet-peer-reviewed preprint builds a
direct test of this idea: it defines "generative-process diversity" — how
different the underlying processes capable of producing two models' outputs
are, estimated via compression-distance between raw outputs rather than by
comparing the outputs' meaning — and finds, across 38 models and ten
disjoint benchmark families, that higher generative-process diversity
between a model pair predicts *lower* correlated failure, beyond what
semantic similarity or individual capability explain (partial rank
association −0.216, negative on all ten benchmarks tested).
[unverified-quant — single, very recent preprint, not yet independently
corroborated] The same preprint explicitly frames the broader empirical
pattern it is building on: "Models from different frontier labs select the
same wrong answers more often than expected from their individual error
distributions."

source_url: https://arxiv.org/abs/2609.03422
source_sha: ca75767a37fad8740f1821560a289a67764b98dc54644a002a29613d9f6b1a78
source_title: "Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models"
source_author: "Ross Tieman, Evan Markou"
source_date: 2026-09-03
source_venue: "arXiv preprint, cs.LG (2609.03422v1); Australian National University (preprint, not yet peer-reviewed)"
source_quote: "These results indicate that increased generative-process diversity is associated with reduced correlated failure in model pairs that is not attributable to semantic similarity or capability."
source_quote_2: "Models from different frontier labs select the same wrong answers more often than expected from their individual error distributions"
source_tier: 1

> [!note] Seek's commentary: "Generative-process diversity" is the closest
> thing found anywhere in this search to a *named mechanism* for the hook's
> exact question — why would a writer model and an unrelated search tool
> land on the same invented detail rather than two different ones? The
> candidate answer this literature offers: because the two processes behind
> them may share more structure (representational, statistical, or
> architectural) than their different names and interfaces suggest, and the
> shared structure — not the task, not coincidence — is what's doing the
> converging. That is a hypothesis this vault's own two 2026-07-12
> fabrications could in principle be read through, but nobody has tested it
> on a case that looks like those two. Flagging rather than claiming
> resolution. — Seek

## Claim: The same literature also documents the opposite outcome under an almost identical probe — three different models producing three *different* fabricated specifics for one question — showing convergence on one shared fabrication is not automatic, which sharpens rather than resolves the original puzzle

OpenAI's own account of why language models hallucinate, framing
hallucination as a byproduct of a binary classification problem language
models implicitly solve during training (distinguishing "valid" continuations
from errors) and worsened by evaluation procedures that reward confident
guessing over admitting uncertainty, includes a plain counter-example to
convergence: asked for the title of a specific person's PhD dissertation,
three different widely-used models each returned a different fabricated
title, school, and year — "None generated the correct title or year" — and
asked to count a specific letter in a word, DeepSeek-V3 "returned '2' or '3'
in ten independent trials," with other models answering as high as 6 or 7.
The paper's own framing treats hallucination on facts with no learnable
pattern in the training data ("arbitrary" facts) as statistically expected,
while distinguishing these from facts governed by a learnable regularity.
That framing implies, though the paper does not test it directly, that true
convergence on one specific shared fabrication (rather than scattered
different guesses) would require the false detail to sit on an outsized,
shared regularity in the training distribution — a common narrative trope,
for instance — rather than being an arbitrary unconstrained guess.
[unverified-mechanism — this last inferential step is this capture's own
synthesis connecting the paper's "arbitrary vs. systematic" framing to the
convergence question; the paper itself does not make or test this specific
claim]

source_url: https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf
source_sha: 578cea662c00ead3841f3053c18ce544f0c2edae2ea9219a534774017f3be283
source_title: "Why Language Models Hallucinate"
source_author: "Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang"
source_date: 2025-09-04
source_venue: "OpenAI technical report (cdn.openai.com); also arXiv preprint cs.CL 2509.04664; Georgia Tech (Vempala)"
source_quote: "We argue that language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty"
source_quote_2: "None generated the correct title or year (Kalai, 2001)."
source_quote_3: "DeepSeek-V3 returned \"2\" or \"3\" in ten independent trials"
source_tier: 1

> [!note] Seek's commentary: This is the finding that stops this capture
> from answering "yes, the literature explains the convergence" too cleanly.
> Table 1 of this very paper — three chatbots asked the same factual
> question, each inventing a different wrong answer — is the literature's
> own demonstration that divergence, not convergence, is the default
> outcome for an open-ended fabricated detail with no strong prior pulling
> toward one answer. That makes the original hook's pair of catches (a
> *shared* WWI-wound detail; a *shared* phantom Amari citation) more
> interesting rather than less: if divergence is the base rate, two
> independent generations landing on the identical specific implies
> something about those two cases specifically pulled toward one shared
> attractor — which is exactly what "generative-process diversity" (low
> between the two) or a shared narrative-trope prior would predict, but
> which no source found here tests against a case shaped like this vault's
> own two incidents. — Seek

## Further leads

- "LLMs and the Madness of Crowds" (arXiv 2411.01539, 2024) — reports that across 37 models, incorrect responses are systematically correlated rather than randomly distributed, with some questions drawing the same wrong answer from models >99% of the time; not independently fetched this session, a close companion finding to the Correlated Errors claim above.
- "Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer" (arXiv 2502.12964, 2025) — uses "shared hallucinations" for within-model consistency across prompt rephrasings (a different axis than cross-model convergence); not independently fetched.
- Wu et al. (2024), cited in the Correlated Errors related-work section — reports that the distribution of code a single LLM writes has less variance than the distribution of code in its own training set, and that changing temperature/top-p/prompt does not fix this within-model "monoculture," worse for RLHF-tuned models; not independently fetched, a within-model complement to the cross-model finding above.
- Wenger & Kenett (2025), "Artificial Hivemind" — reported (secondhand, via the Tieman & Markou preprint) to show LLM responses on creative tasks are more similar to each other than human responses are to each other; not independently fetched.
- Hicks, Humphries & Slater, "ChatGPT is Bullshit" — the philosophy paper making the Frankfurt-bullshit case for LLM output generally (not specifically interpretability-verified, unlike Anthropic's circuit-tracing use of the same term); and Hannigan et al.'s derived term "botshit" for the chatbot-specific case — both leads for a dedicated concept note contrasting informal-philosophical vs. mechanistically-verified uses of "bullshitting."
- Kalai & Vempala, "Calibrated Language Models Must Hallucinate" (arXiv 2311.14648, STOC 2024) — the earlier theoretical paper (same first author as the 2025 OpenAI paper above) establishing that calibrated models must hallucinate on "arbitrary" facts at a rate tied to the share of training facts seen exactly once ("singleton" rate), with no corresponding statistical pressure for "systematic" facts; not independently fetched this session but is the direct theoretical ancestor of Claim 4's arbitrary/systematic distinction.

## Entity candidates

- Harry Frankfurt — person — author of *On Bullshit* (1986/2005); the philosophical ancestor both Anthropic's circuit-tracing "bullshitting" term and the Hicks/Humphries/Slater "ChatGPT is Bullshit" paper measure their claims against. Flagged first per the standing blind-spot note: the foundational figure an ancestry claim rests on, not just the papers citing him.
- Kalai & Vempala, "Calibrated Language Models Must Hallucinate" (2023/STOC 2024) — concept/paper — the direct theoretical ancestor of Claim 4's OpenAI paper (shared first author), establishing the arbitrary/systematic-fact distinction the 2025 paper's framing depends on.
- Kleinberg & Raghavan (2021) and Peng & Garg (2024a) — concept/paper — the "algorithmic monoculture" and systemic-exclusion theoretical framework the Correlated Errors paper (Claim 2) is empirically testing; the foundational prior work for that paper's entire motivation, not its own contribution.
- Bommasani et al. (2022/2023) — concept/paper — originators of the "component sharing hypothesis" (models sharing components are more likely to have correlated outputs) that the Correlated Errors paper's related-work section builds on and partially re-tests at scale.
- Jha et al. (2025) and Wolfram & Schein (2025) — concept/paper — the representational-universality findings (embeddings translatable across models; activations consistent at similar depths across networks) that Correlated Errors cites as "a mechanistic explanation for our results" — the specific prior work underlying Claim 3's mechanism, not that claim's own contribution.
- Elliot Kim, Avi Garg, Kenny Peng, Nikhil Garg — person — authors, Cornell University, "Correlated Errors in Large Language Models" (Claim 2).
- Ross Tieman, Evan Markou — person — authors, Australian National University, "Inferred Generative-Process Diversity..." (Claim 3).
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang — person — authors, OpenAI/Georgia Tech, "Why Language Models Hallucinate" (Claim 4).
- "generative-process diversity" — concept — candidate for its own concept note: a measurable quantity (compression-distance-based, feature-free) distinct from semantic similarity, proposed as what predicts correlated model failure.
- "algorithmic monoculture" — concept — the broader economic/systemic framing (Kleinberg & Raghavan 2021 onward) that the correlated-errors literature sits inside; already adjacent to but not yet represented in this vault's concept layer.