Is Petiška's 2023 finding that GPT-4 selects citations by raw Google Scholar popularity replicated elsewhere, and does the vault's citational-narrowing pattern predict how that bias compounds as LLM output becomes training data?

chatgptllmcitation-metricsmatthew-effectbibliometricsgoogle-scholarmodel-collapsefeedback-loopreplicationrobert-merton

This capture researches the topic hooked from observation-petiska-chatgpt-matthew-effect-bridges-garfield-warning-and-rag-reliability, which established that Petiška's 2023 single-author preprint is the missing middle term between Garfield's citation-metrics warning and the vault's reliability-aware-RAG cluster. That existing note already flags itself as "a first look rather than a confirmed finding" and asks for "a peer-reviewed or larger-N replication." This capture went looking for exactly that, and for evidence bearing on whether the bias could compound as LLM output re-enters training corpora.

Claim: Petiška's finding is independently replicated by a differently-authored, peer-reviewed study using a different citation database, different field, and a different task design

Claim type: quantitative + technical-mechanism. Tier 1-2 required — met (arXiv preprint, later peer-reviewed).

Algaba, Mazijn, Holst, Tori, Wenmackers & Ginis (Vrije Universiteit Brussel, KU Leuven, Harvard) tasked GPT-4 with reconstructing 3,066 anonymized in-text citations across 166 machine-learning papers (AAAI, NeurIPS, ICML, ICLR) published after GPT-4's training cutoff, verifying existence and metadata against Semantic Scholar rather than Google Scholar. Their result: "GPT-4 exhibits strong preferences for highly cited papers, which persists even after controlling for multiple confounding factors such as publication year, title length, venue, and number of authors." The median citation-count gap between GPT-4's existing generated references and the ground-truth references they were meant to reconstruct was 1,326 (and 1,257 when controlling for recency); the bias held across every title length, author count, and venue bucket tested. Unlike Petiška's own non-peer-reviewed preprint, this paper was subsequently published as "Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias," Findings of the Association for Computational Linguistics: NAACL 2025, pp. 6829-6864 — a fact independently confirmed in a separate paper's own reference list (He 2025/2026, "Who Gets Cited?," reference entry: "Algaba, A.; Mazijn, C.; Holst, V.; Tori, F.; Wenmackers, S.; and Ginis, V. 2025... In Findings of the Association for Computational Linguistics: NAACL 2025, 6829-6864... ISBN 979-8-89176-195-7").

This is a genuinely independent replication: different author group, no citation of Petiška anywhere in its text, a different underlying database (Semantic Scholar, not Google Scholar), a different task (reconstructing citations for an existing paper's introduction, not generating new literature-review text), and a different field (computer science, not environmental science) — yet the same core result, that GPT-4's citation selection skews toward already-highly-cited work regardless of relevance or recency, described in the paper's own words as a mechanism that "may also amplify existing biases and introduce new ones, potentially skewing scientific knowledge dissemination."

Claim: A second, later, and much larger-scale study across ten current-generation commercial LLMs — explicitly citing Petiška — confirms the same popularity-driven citation-selection bias generalizes well beyond GPT-4

Claim type: quantitative. Tier 1-2 required — met (arXiv preprint; single-author, unrefereed at time of capture — see caveat below).

M.Z. Naser (Clemson University) audited 69,557 citation instances generated by ten commercially deployed LLMs (GPT-5-mini, GPT-5-nano, GPT-4o-mini, Claude haiku-3.5, Claude haiku-4.5, Llama4-scout, Llama4-maverick, DeepSeek-v3.1, Kimi-k2.5, Mistral-small-3) across four academic domains, verifying references against CrossRef, OpenAlex, and Semantic Scholar. The paper cites Petiška's 2023 paper directly as prior work (its reference [5]: "E. Petiska, ChatGPT cites the most-cited articles and journals, relying solely on Google Scholar's citation counts. As a result, AI may amplify the Matthew Effect in environmental science, ArXiv (Cornell University). (2023)"). Its own finding, on citation-count popularity specifically: "median cited-by counts for confirmed references range from 359 (kimi-k2.5) to 1,132 (GPT-5-nano), while field-level medians for the four domains in our study fall between 50 and 100 citations. This implies that LLMs do not seem to sample uniformly from their training distributions, but instead they preferentially retrieve highly cited works (which further amplifies the visibility of already-prominent research)." The bias was strongest in the two most accurate (lowest-hallucination) models tested, GPT-5-mini and GPT-5-nano, at p < 10⁻⁴⁶.

Together with the Algaba et al. replication above, this establishes that Petiška's original single-model, single-field 2023 finding is not an isolated result: the same popularity-driven selection mechanism has now been independently measured across at least three author groups, multiple citation databases, multiple academic fields, and — as of this second study — across the current generation of frontier models from every major vendor (OpenAI, Anthropic, Meta, DeepSeek, Moonshot AI, Mistral), roughly three years after Petiška's original paper.

Caveat carried forward, not silently dropped: Naser 2026 is, like Petiška's own paper, a single-author preprint not yet independently peer-reviewed at the time of this capture — it does not itself discharge the single-source concentration cap, it is a second, separate unrefereed primary. Algaba et al. is the one leg of this claim that clears peer review.

Claim: A documented case exists of an LLM-generated citation artifact propagating from one AI-assisted publication into another's reference list — direct evidence of the compounding mechanism, though for citation fabrication rather than popularity-selection specifically, and observed once

Claim type: technical-mechanism. Tier 1-2 required — met (arXiv preprint, single case, quoted directly; flagged below as n=1).

Samar Ansari (University of Chester) analyzed 100 hallucinated citations found (via GPTZero's automated tooling) in 53 NeurIPS 2025 accepted papers. Within that sample, one citation — "Z. Zhu, T. Yu, X. Zhang, J. Li, Y. Zhang, and Y. Fu. Neuralrgb-d..." — traced back to an earlier arXiv preprint (Beltran et al., v1, arXiv:2412.13176) that had contained the identical fabricated citation before a later version corrected it. Ansari concludes: "This suggests the hallucination may not have originated with the NeurIPS author's LLM but was instead inherited from contaminated training data. The language model likely encountered the erroneous citation in Beltran et al. (v1), learned it as a valid pattern, and reproduced it... We have named this failure mode as 'Contamination Inheritance (CI).'" The paper explicitly frames this as evidence of "a contamination feedback loop where AI-generated errors are learned by subsequent models," and calls for "training-data audits, version-controlled corpus tracking, and citation genealogy mapping" to determine whether Contamination Inheritance is "already widespread or represents isolated cases" — an open question in Ansari's own telling, not a closed one.

This is the clearest documented instance found of the general mechanism the topic asks about — one system's output entering the corpus and being reproduced by a later system — but it differs from Petiška's specific popularity-driven mechanism in kind (fabricated-content propagation, not citation-count-driven selection) and in scale (one traced case within a 100-citation sample, not a systematic multi-generation study).

Verdict on the core question

Replication: confirmed. Two independent studies (Algaba et al. 2025, peer-reviewed; Naser 2026, preprint) reproduce Petiška's core finding — LLM citation selection skews toward already-highly-cited work — using different models, databases, fields, and task designs, one of them explicitly citing Petiška as its starting point.

Compounding through training data: [unverified — could not confirm or deny at scale after genuine search]. No study located runs a multi- generation retraining or iterated-learning experiment that empirically measures whether citation-popularity bias specifically gets worse across model generations as LLM-selected citations re-enter training corpora. What was found: a named, real, singly-documented mechanism by which an AI-generated citation artifact propagates from one model's output into what a later model reproduces (Ansari's Contamination Inheritance), and an explicit but hedged theoretical concern from a source already established as Tier 1 on this topic (Naser 2026: "As LLMs become more integrated into literature review workflows, this training-data-mediated bias could compound existing disparities in citation patterns"). The vault's own Matthew Effect framing — credit compressing toward whoever already has it, the same pattern already documented in the vault for the backpropagation citation gap — describes exactly the shape a training-data feedback loop would take if it existed, and supplies a testable prediction (repeated training rounds should narrow the cited set further toward whoever is already popular). But a prediction being theoretically well-formed and independently plausible is not the same as it being empirically confirmed; that experiment was not found.

Further leads

• He, J. (2025/2026), "Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection" (arXiv:2508.02740) — a distinct bias axis (author gender/majority-group, not citation-count popularity) in the same reference-selection pipeline; GPT-4o shows persistent male-favoring bias, Claude models show majority-favoring bias instead; not folded into the claims above because it is a different mechanism, but directly adjacent and explicitly names the training-data compounding risk quoted in this capture's frontmatter.
• Tian, Y. et al. (2024/2025), "Who Gets Recommended? Investigating Gender, Race, and Country Disparities in Paper Recommendations from Large Language Models" (arXiv:2501.00367) — cited by He 2025 as confirming that LLM-generated recommendations mirror real-world team size and favor incremental over disruptive research; not independently read this session.
• GPTZero's own NeurIPS 2025 hallucination report (gptzero.me/news/neurips) and ICLR 2026 submissions report (gptzero.me/news/iclr-2026) — the underlying third-party data source Ansari 2026 built its analysis on; not independently verified this session, only read through Ansari's citations of it.
• Schaeffer et al. 2025 and the Nature 2024 iterated-learning comment (already in the vault's model-collapse cluster, per 50-questions/_answered/question-verify-schaeffer-2025-eight-model-collapse-definitions.md and question-verify-nature-2024-iterated-learning-model-collapse-comment.md) — the general recursive-training-degrades-quality literature exists in depth already in this vault; this capture deliberately did not re-derive it, since the gap is specifically citation-popularity bias compounding, not model collapse in general.
• Walters & Wilder 2023 (Scientific Reports, "Fabrication and errors in the bibliographic citations generated by ChatGPT") — the earliest quantitative baseline cited by both Naser 2026 and other sources found this session (55% of GPT-3.5 citations, 18% of GPT-4 citations entirely fabricated); worth a direct read as the origin point of the citation-fabrication-rate literature, distinct from the popularity-bias literature this capture focused on.

Entity candidates

• Robert Merton — person — the foundational figure this entire cluster measures itself against; both replications found this session (Algaba et al., Naser) invoke "the Matthew effect" or Merton by name as the phenomenon their citation-count findings instantiate. Already an entity in this vault (entity-robert-merton); flagged first per this capture's own instructions, not to duplicate the page.
• Eugene Garfield — person — already an entity (entity-eugene-garfield); the citation-metrics-skepticism figure this whole cluster (Petiška, Algaba, Naser, Ansari) sits downstream of, per the existing bridge observation.
• Andres Algaba — person — first author of the peer-reviewed replication (Claim 1); no entity page exists yet.
• M.Z. Naser — person — author of the larger ten-model replication (Claim 2) and its explicit citation of Petiška; no entity page exists yet.
• Samar Ansari — person — author of the Contamination Inheritance finding (Claim 3); no entity page exists yet.
• Contamination Inheritance — concept/term — the named failure mode from Ansari 2026 describing an LLM reproducing a citation error it inherited from prior contaminated training data; the closest documented real-world instance of the compounding mechanism this capture's core question asks about.
• Eduard Petiška — person — subject of the vault's existing claim-note; no entity page exists yet despite being the anchor of an entire vault cluster (Garfield bridge, RA-RAG bridge, this capture).