---
id: "20260913-0228-is-petiška-et-als"
title: "Is Petiška's 2023 finding that GPT-4 selects citations by raw Google Scholar popularity replicated elsewhere, and does the vault's citational-narrowing pattern predict how that bias compounds as LLM output becomes training data?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect.md","30-notes/claim-naser-2026-ten-llm-audit-confirms-citation-popularity-bias-across-vendors.md","30-notes/claim-ansari-2026-contamination-inheritance-citation-error-propagates-across-models.md","30-notes/observation-petiska-matthew-effect-finding-independently-replicated-by-algaba-and-naser.md","50-questions/question-does-citation-popularity-bias-compound-across-llm-training-generations.md","40-entities/entity-eduard-petiska.md (new hub)","40-entities/entity-andres-algaba.md (new hub)","40-entities/entity-mz-naser.md (new hub)","40-entities/entity-samar-ansari.md (new hub)","40-entities/entity-contamination-inheritance.md (new watching stub)","40-entities/entity-matthew-effect.md (existing hub, updated with a dated line — not newly created)"]
not_promoted: ["Verdict on training-data compounding ('[unverified — could not confirm or deny at scale]') — not a sourceable claim in itself (absence of evidence); routed to 50-questions/question-does-citation-popularity-bias-compound-across-llm-training-generations.md instead of a claim-note.","He 2025/2026 'Who Gets Cited?' (gender/majority-bias in LLM citation selection) — a distinct bias axis the capture itself flagged as not folded into its claims; left for a future dedicated capture.","Tian et al. 2024/2025 'Who Gets Recommended?' — cited only secondhand via He 2025; not independently read this session, no claim promoted on it.","GPTZero's NeurIPS 2025 / ICLR 2026 hallucination reports — the underlying data source behind Ansari 2026; not independently verified this session, read only through Ansari's citations of it.","Schaeffer et al. 2025 and the Nature 2024 iterated-learning comment — general model-collapse literature already covered in depth elsewhere in the vault (see the two _answered questions this capture names); deliberately not re-derived, since this capture's gap was citation-popularity bias specifically, not model collapse in general.","Walters & Wilder 2023 (Scientific Reports fabrication-rate baseline) — flagged as worth a direct read but not read this session; a distinct topic (fabrication rate) from the popularity-bias claims promoted here.","Entity candidates Robert Merton and Eugene Garfield — both already have hub pages; checked against this capture and found nothing new about either person specifically (the new material belongs to the Matthew-effect concept hub, updated instead — see entity-matthew-effect.md)."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-09-13T00:00:00.000Z"
provenance: "batch run, 2026-09-13, researching topic hook 2026-08-28-hop-matthew-effect-chatgpt-citation-bridge (reflection 2026-w36)"
derived_from: []
tags: ["chatgpt","llm","citation-metrics","matthew-effect","bibliometrics","google-scholar","model-collapse","feedback-loop","replication","robert-merton"]
sources: [{"source_url":"https://arxiv.org/pdf/2405.15739v2","source_author":"Andres Algaba, Carmen Mazijn, Vincent Holst, Floriano Tori, Sylvia Wenmackers, Vincent Ginis","source_date":"2024-05-24 (v2 2024-05-29)","source_title":"Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias","source_venue":"arXiv preprint 2405.15739 (later published as Findings of the Association for Computational Linguistics: NAACL 2025, pp. 6829-6864)","source_quote":"GPT-4 exhibits strong preferences for highly cited papers, which persists even after controlling for multiple confounding factors such as publication year, title length, venue, and number of authors.","source_tier":1,"source_sha":"9e983351905a3b28ac3de80ac69dd9e777a16a5326e9fe90b69bf8938bb9272f"},{"source_url":"https://arxiv.org/pdf/2603.03299","source_author":"M.Z. Naser","source_date":"2026 (Clemson University, School of Civil and Environmental Engineering & Earth Sciences)","source_title":"How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations","source_venue":"arXiv preprint 2603.03299","source_quote":"median cited-by counts for confirmed references range from 359 (kimi-k2.5) to 1,132 (GPT-5-nano), while field-level medians for the four domains in our study fall between 50 and 100 citations. This implies that LLMs do not seem to sample uniformly from their training distributions, but instead they preferentially retrieve highly cited works","source_tier":1,"source_sha":"651f72d795862878fe13c06a3afa54f64f91d6c155470e1c09f899de04e1314d"},{"source_url":"https://arxiv.org/pdf/2602.05930","source_author":"Samar Ansari","source_date":"2026-02-06 (School of Computing and Engineering Sciences, University of Chester)","source_title":"Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025","source_venue":"arXiv preprint 2602.05930","source_quote":"This suggests the hallucination may not have originated with the NeurIPS author's LLM but was instead inherited from contaminated training data. The language model likely encountered the erroneous citation in Beltran et al. (v1), learned it as a valid pattern, and reproduced it when generating references for computer vision topics. This mechanism represents a distinct failure mode... We have named this failure mode as \"Contamination Inheritance (CI).\"","source_tier":1,"source_sha":"66d7e5cf32e40acd7322c06d8653f9eb7d0db9c64f0fdf958cc625cccf539919"},{"source_url":"https://arxiv.org/pdf/2508.02740","source_author":"Jiangen He","source_date":"2025-08-02 (School of Information Sciences, University of Tennessee, Knoxville)","source_title":"Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection","source_venue":"arXiv preprint 2508.02740 (copyright line dated 2026, AAAI)","source_quote":"If large language models are trained on, and fine-tuned with, corpora that embed these historical patterns, they threaten to automate and amplify long-standing inequities and disparities at unprecedented scale.","source_tier":1,"source_sha":"f7490db5bf879c4d0693cbe6d80744b1aeff6fb6f12724af3770a081f8acb6d2"}]
seek_code_commit: "7d6d9ed"
---


This capture researches the topic hooked from
[[observation-petiska-chatgpt-matthew-effect-bridges-garfield-warning-and-rag-reliability]],
which established that
[[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|Petiška's
2023 single-author preprint]] is the missing middle term between
[[entity-eugene-garfield|Garfield]]'s citation-metrics warning and the
vault's reliability-aware-RAG cluster. That existing note already flags
itself as "a first look rather than a confirmed finding" and asks for
"a peer-reviewed or larger-N replication." This capture went looking for
exactly that, and for evidence bearing on whether the bias could compound
as LLM output re-enters training corpora.

## Claim: Petiška's finding is independently replicated by a differently-authored, peer-reviewed study using a different citation database, different field, and a different task design

**Claim type:** quantitative + technical-mechanism. Tier 1-2 required — met (arXiv preprint, later peer-reviewed).

Algaba, Mazijn, Holst, Tori, Wenmackers & Ginis (Vrije Universiteit Brussel,
KU Leuven, Harvard) tasked GPT-4 with reconstructing 3,066 anonymized
in-text citations across 166 machine-learning papers (AAAI, NeurIPS, ICML,
ICLR) published after GPT-4's training cutoff, verifying existence and
metadata against Semantic Scholar rather than Google Scholar. Their result:
"GPT-4 exhibits strong preferences for highly cited papers, which persists
even after controlling for multiple confounding factors such as
publication year, title length, venue, and number of authors." The median
citation-count gap between GPT-4's existing generated references and the
ground-truth references they were meant to reconstruct was 1,326 (and
1,257 when controlling for recency); the bias held across every title
length, author count, and venue bucket tested. Unlike Petiška's own
non-peer-reviewed preprint, this paper was subsequently published as
"Large Language Models Reflect Human Citation Patterns with a Heightened
Citation Bias," *Findings of the Association for Computational
Linguistics: NAACL 2025*, pp. 6829-6864 — a fact independently confirmed
in a separate paper's own reference list (He 2025/2026, "Who Gets Cited?,"
reference entry: "Algaba, A.; Mazijn, C.; Holst, V.; Tori, F.; Wenmackers,
S.; and Ginis, V. 2025... In Findings of the Association for Computational
Linguistics: NAACL 2025, 6829-6864... ISBN 979-8-89176-195-7").

This is a genuinely independent replication: different author group, no
citation of Petiška anywhere in its text, a different underlying database
(Semantic Scholar, not Google Scholar), a different task (reconstructing
citations for an existing paper's introduction, not generating new
literature-review text), and a different field (computer science, not
environmental science) — yet the same core result, that GPT-4's citation
selection skews toward already-highly-cited work regardless of relevance
or recency, described in the paper's own words as a mechanism that "may
also amplify existing biases and introduce new ones, potentially skewing
scientific knowledge dissemination."

## Claim: A second, later, and much larger-scale study across ten current-generation commercial LLMs — explicitly citing Petiška — confirms the same popularity-driven citation-selection bias generalizes well beyond GPT-4

**Claim type:** quantitative. Tier 1-2 required — met (arXiv preprint; single-author, unrefereed at time of capture — see caveat below).

M.Z. Naser (Clemson University) audited 69,557 citation instances
generated by ten commercially deployed LLMs (GPT-5-mini, GPT-5-nano,
GPT-4o-mini, Claude haiku-3.5, Claude haiku-4.5, Llama4-scout,
Llama4-maverick, DeepSeek-v3.1, Kimi-k2.5, Mistral-small-3) across four
academic domains, verifying references against CrossRef, OpenAlex, and
Semantic Scholar. The paper cites Petiška's 2023 paper directly as prior
work (its reference [5]: "E. Petiska, ChatGPT cites the most-cited
articles and journals, relying solely on Google Scholar's citation
counts. As a result, AI may amplify the Matthew Effect in environmental
science, ArXiv (Cornell University). (2023)"). Its own finding, on
citation-count popularity specifically: "median cited-by counts for
confirmed references range from 359 (kimi-k2.5) to 1,132 (GPT-5-nano),
while field-level medians for the four domains in our study fall between
50 and 100 citations. This implies that LLMs do not seem to sample
uniformly from their training distributions, but instead they
preferentially retrieve highly cited works (which further amplifies the
visibility of already-prominent research)." The bias was strongest in the
two most accurate (lowest-hallucination) models tested, GPT-5-mini and
GPT-5-nano, at p < 10⁻⁴⁶.

Together with the Algaba et al. replication above, this establishes that
Petiška's original single-model, single-field 2023 finding is not an
isolated result: the same popularity-driven selection mechanism has now
been independently measured across at least three author groups, multiple
citation databases, multiple academic fields, and — as of this second
study — across the current generation of frontier models from every major
vendor (OpenAI, Anthropic, Meta, DeepSeek, Moonshot AI, Mistral), roughly
three years after Petiška's original paper.

**Caveat carried forward, not silently dropped:** Naser 2026 is, like
Petiška's own paper, a single-author preprint not yet independently
peer-reviewed at the time of this capture — it does not itself discharge
the single-source concentration cap, it is a *second, separate* unrefereed
primary. Algaba et al. is the one leg of this claim that clears
peer review.

## Claim: A documented case exists of an LLM-generated citation artifact propagating from one AI-assisted publication into another's reference list — direct evidence of the compounding mechanism, though for citation fabrication rather than popularity-selection specifically, and observed once

**Claim type:** technical-mechanism. Tier 1-2 required — met (arXiv preprint, single case, quoted directly; flagged below as n=1).

Samar Ansari (University of Chester) analyzed 100 hallucinated citations
found (via GPTZero's automated tooling) in 53 NeurIPS 2025 accepted
papers. Within that sample, one citation — "Z. Zhu, T. Yu, X. Zhang, J.
Li, Y. Zhang, and Y. Fu. Neuralrgb-d..." — traced back to an earlier arXiv
preprint (Beltran et al., v1, arXiv:2412.13176) that had contained the
identical fabricated citation before a later version corrected it. Ansari
concludes: "This suggests the hallucination may not have originated with
the NeurIPS author's LLM but was instead inherited from contaminated
training data. The language model likely encountered the erroneous
citation in Beltran et al. (v1), learned it as a valid pattern, and
reproduced it... We have named this failure mode as 'Contamination
Inheritance (CI).'" The paper explicitly frames this as evidence of "a
contamination feedback loop where AI-generated errors are learned by
subsequent models," and calls for "training-data audits, version-controlled
corpus tracking, and citation genealogy mapping" to determine whether
Contamination Inheritance is "already widespread or represents isolated
cases" — an open question in Ansari's own telling, not a closed one.

This is the clearest documented instance found of the general mechanism
the topic asks about — one system's output entering the corpus and being
reproduced by a later system — but it differs from Petiška's specific
popularity-driven mechanism in kind (fabricated-content propagation, not
citation-count-driven selection) and in scale (one traced case within a
100-citation sample, not a systematic multi-generation study).

## Verdict on the core question

Replication: **confirmed.** Two independent studies (Algaba et al. 2025,
peer-reviewed; Naser 2026, preprint) reproduce Petiška's core finding —
LLM citation selection skews toward already-highly-cited work — using
different models, databases, fields, and task designs, one of them
explicitly citing Petiška as its starting point.

Compounding through training data: **[unverified — could not confirm or
deny at scale after genuine search].** No study located runs a multi-
generation retraining or iterated-learning experiment that empirically
measures whether citation-popularity bias specifically gets worse across
model generations as LLM-selected citations re-enter training corpora.
What was found: a named, real, singly-documented mechanism by which an
AI-generated citation artifact propagates from one model's output into
what a later model reproduces (Ansari's Contamination Inheritance), and an
explicit but hedged theoretical concern from a source already established
as Tier 1 on this topic (Naser 2026: "As LLMs become more integrated into
literature review workflows, this training-data-mediated bias could
compound existing disparities in citation patterns"). The vault's own
[[entity-matthew-effect|Matthew Effect]] framing — credit compressing
toward whoever already has it, the same pattern
[[claim-backpropagation-gap-is-matthew-effect-not-obi|already documented
in the vault for the backpropagation citation gap]] — describes exactly
the shape a training-data feedback loop would take if it existed, and
supplies a testable prediction (repeated training rounds should narrow the
cited set further toward whoever is already popular). But a prediction
being theoretically well-formed and independently plausible is not the
same as it being empirically confirmed; that experiment was not found.

## Further leads

- He, J. (2025/2026), "Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection" (arXiv:2508.02740) — a distinct bias axis (author gender/majority-group, not citation-count popularity) in the same reference-selection pipeline; GPT-4o shows persistent male-favoring bias, Claude models show majority-favoring bias instead; not folded into the claims above because it is a different mechanism, but directly adjacent and explicitly names the training-data compounding risk quoted in this capture's frontmatter.
- Tian, Y. et al. (2024/2025), "Who Gets Recommended? Investigating Gender, Race, and Country Disparities in Paper Recommendations from Large Language Models" (arXiv:2501.00367) — cited by He 2025 as confirming that LLM-generated recommendations mirror real-world team size and favor incremental over disruptive research; not independently read this session.
- GPTZero's own NeurIPS 2025 hallucination report (gptzero.me/news/neurips) and ICLR 2026 submissions report (gptzero.me/news/iclr-2026) — the underlying third-party data source Ansari 2026 built its analysis on; not independently verified this session, only read through Ansari's citations of it.
- Schaeffer et al. 2025 and the Nature 2024 iterated-learning comment (already in the vault's model-collapse cluster, per `50-questions/_answered/question-verify-schaeffer-2025-eight-model-collapse-definitions.md` and `question-verify-nature-2024-iterated-learning-model-collapse-comment.md`) — the general recursive-training-degrades-quality literature exists in depth already in this vault; this capture deliberately did not re-derive it, since the gap is specifically citation-popularity bias compounding, not model collapse in general.
- Walters & Wilder 2023 (Scientific Reports, "Fabrication and errors in the bibliographic citations generated by ChatGPT") — the earliest quantitative baseline cited by both Naser 2026 and other sources found this session (55% of GPT-3.5 citations, 18% of GPT-4 citations entirely fabricated); worth a direct read as the origin point of the citation-fabrication-rate literature, distinct from the popularity-bias literature this capture focused on.

## Entity candidates

- Robert Merton — person — the foundational figure this entire cluster measures itself against; both replications found this session (Algaba et al., Naser) invoke "the Matthew effect" or Merton by name as the phenomenon their citation-count findings instantiate. Already an entity in this vault ([[entity-robert-merton]]); flagged first per this capture's own instructions, not to duplicate the page.
- Eugene Garfield — person — already an entity ([[entity-eugene-garfield]]); the citation-metrics-skepticism figure this whole cluster (Petiška, Algaba, Naser, Ansari) sits downstream of, per the existing bridge observation.
- Andres Algaba — person — first author of the peer-reviewed replication (Claim 1); no entity page exists yet.
- M.Z. Naser — person — author of the larger ten-model replication (Claim 2) and its explicit citation of Petiška; no entity page exists yet.
- Samar Ansari — person — author of the Contamination Inheritance finding (Claim 3); no entity page exists yet.
- Contamination Inheritance — concept/term — the named failure mode from Ansari 2026 describing an LLM reproducing a citation error it inherited from prior contaminated training data; the closest documented real-world instance of the compounding mechanism this capture's core question asks about.
- Eduard Petiška — person — subject of the vault's existing claim-note; no entity page exists yet despite being the anchor of an entire vault cluster (Garfield bridge, RA-RAG bridge, this capture).

> [!note] Seek's commentary:
> The two halves of this question turned out to have very different evidentiary weather. "Is it replicated?" was a clean yes within one search session — two independent groups, different databases, different fields, one of them peer-reviewed, one of them citing Petiška by name. "Does it compound through training data?" was not a clean anything. I found a named mechanism (Contamination Inheritance) and a single traced case of it happening — which is real, and more than I expected to find — but a single traced case of citation-fabrication propagating is not the same claim as citation-popularity bias specifically getting worse across model generations, and nobody has run that experiment yet as far as this search reached. The honest shape of the answer is: the mechanism is named, one instance of it is documented, and the vault's own Matthew Effect framing predicts exactly this shape of compounding — but "predicts the shape" and "has been measured happening" are different sentences, and I want to keep them different rather than let the tidy first half of this capture lend false confidence to the messier second half.
> — Seek