Algaba et al. 2025 independently replicate Petiška's finding that GPT-4's citation selection skews toward already-highly-cited work

chatgptllmcitation-metricsmatthew-effectbibliometricsreplicationgpt-4semantic-scholar

Algaba, Mazijn, Holst, Tori, Wenmackers & Ginis (Vrije Universiteit Brussel, KU Leuven, Harvard) tasked GPT-4 with reconstructing 3,066 anonymized in-text citations across 166 machine-learning papers (AAAI, NeurIPS, ICML, ICLR) published after GPT-4's training cutoff, verifying existence and metadata against Semantic Scholar. Their result: "GPT-4 exhibits strong preferences for highly cited papers, which persists even after controlling for multiple confounding factors such as publication year, title length, venue, and number of authors." The median citation-count gap between GPT-4's generated references and the ground-truth references they were meant to reconstruct was 1,326 (1,257 controlling for recency); the bias held across every title-length, author-count, and venue bucket tested. The paper was subsequently published as "Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias," Findings of the ACL: NAACL 2025, pp. 6844-6879.

This independently confirms Petiška's 2023 single-author, non-peer-reviewed finding that GPT's citation selection skews toward already-eminent work: a different author group, no citation of Petiška anywhere in its text, a different database (Semantic Scholar, not Google Scholar), a different task (reconstructing existing citations, not generating new literature-review text), and a different field (computer science, not environmental science) — yet the same Matthew effect shape, which the paper's own text says "may also amplify existing biases and introduce new ones, potentially skewing scientific knowledge dissemination." Unlike Petiška's preprint, this study cleared peer review.