Does GPT-4's citation-selection bias toward highly-cited papers emerge from the same copying mechanism Simkin & Roychowdhury's 2005 model uses to explain human citation preferential attachment?

citation-copyingbibliometricsmatthew-effectpreferential-attachmentllm-citation-biasmechanism-comparisontraining-data-exposure

Summary of what was found

No source located in this research directly tests the compound causal claim the topic poses — that GPT-4's preference for highly-cited papers is a laundered inheritance of a citation ecosystem Simkin & Roychowdhury's copying model explains as itself produced by copying, rather than an independently-formed preference. The two mechanisms (human reference-list copying; LLM training-data frequency effects) occupy adjacent links of a plausible causal chain, but no paper splices them together: nobody has tested whether the overrepresentation of highly-cited papers in an LLM's training corpus is itself attributable to copying-driven reference-list propagation (Simkin's specific mechanism) as opposed to other routes to citation-count inflation (independent merit, media attention, open-access availability). What does exist, and is captured below, is (1) the GPT-4 citation-bias paper's own explicit hypothesis that the bias is a training-data-frequency effect, stated in general terms without reference to any copying model, and (2) a 2026 paper that directly tests and confirms a structurally identical "inherited from pretraining exposure" mechanism for general entity popularity (not citations). Neither paper invokes or tests against Simkin & Roychowdhury specifically. The central question is marked [unverified — could not confirm or deny after search].

Claim: Algaba et al. (2025) themselves propose training-data citation frequency, not an independently-formed fame preference, as the likely source of GPT-4's citation bias

In the Discussion section of the paper the vault already cites for the bias finding itself (claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect), the authors offer their own explanation for why the bias exists — a mechanism claim distinct from the bias-exists finding already in the vault: "One plausible hypothesis for the heightened citation bias observed in GPT-4 is the increased frequency of citations to heavily cited papers within the model's training data. This prevalence makes these references more likely to be generated accurately and recognized as existent." They extend this to a general training-dynamics account: "such biases may stem from generic training effects, where models preferentially learn patterns that are more common in the data, leading to biases towards shorter titles, more heavily cited, and slightly less recent works."

This is the paper's own stated hypothesis, not a demonstrated causal test — the paper verifies the existence of the bias, not the mechanism producing it; no experiment in the paper isolates training-data citation frequency as a controlled variable. The explanation is framed as plausible, not confirmed, and the paper does not cite or engage Simkin & Roychowdhury or any copying model; "frequency in the training corpus" is offered as a bare fact about the corpus, with no claim about how that frequency distribution itself arose.

• source_url: "https://arxiv.org/pdf/2405.15739v2"
• source_author: "Andres Algaba, Carmen Mazijn, Vincent Holst, Floriano Tori, Sylvia Wenmackers, Vincent Ginis"
• source_date: "2024-05-24 (v2 2024-05-29); published Findings of ACL: NAACL 2025, pp. 6844-6879"
• source_title: "Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias"
• source_venue: "arXiv preprint 2405.15739 (v2); published as Findings of the Association for Computational Linguistics: NAACL 2025 (ACL Anthology 2025.findings-naacl.381)"
• source_quote: "One plausible hypothesis for the heightened citation bias observed in GPT-4 is the increased frequency of citations to heavily cited papers within the model's training data. This prevalence makes these references more likely to be generated accurately and recognized as existent."
• source_tier: 1
• source_sha: "9e983351905a3b28ac3de80ac69dd9e777a16a5326e9fe90b69bf8938bb9272f"
• claim type: technical-mechanism (the paper's own proposed, unconfirmed mechanism) — floor cleared (Tier 1)

Claim: A 2026 study directly confirms that LLM popularity judgments are better explained by pretraining-corpus exposure frequency than by independent real-world popularity signals — for general entities, not citations

Mozafari, Piryani & Jatowt (University of Innsbruck) built the first large-scale test of whether LLM popularity preferences track real-world popularity or statistical exposure during pretraining, using the fully open OLMo models and their complete Dolma pretraining corpus (7.4 trillion tokens, indexed with the Infini-gram toolkit) to compute exact entity-level exposure counts for 2,000 entities across five types (Person, Location, Organization, Art, Product). They compared this ground-truth exposure measure against Wikipedia pageviews (an external, independent popularity signal) and two elicited LLM popularity judgments (direct scalar estimation; pairwise comparison). Their finding: "LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons," an effect "strongest for larger models" that "persists in the long tail, where Wikipedia popularity becomes unreliable." The paper's own conclusion: "popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias."

This is a direct, controlled test of the general mechanism-class the topic question invokes — "is the model's preference an inherited property of what it was trained on, rather than an independently-formed judgment of real-world merit or fame?" — and the answer, for entity popularity broadly, is yes. It is not a citation-specific test: entities are people, places, organizations, artworks, and products, not papers or citation counts, and the paper does not address citation selection or cite Algaba, Simkin, or any bibliometric copying model. It is the closest available empirical analog to the inheritance half of the topic's hypothesis, not a direct test of it.

• source_url: "https://arxiv.org/pdf/2605.12382"
• source_author: "Jamshid Mozafari, Bhawna Piryani, Adam Jatowt"
• source_date: "2026-05-12 (arXiv v1); to appear, Proceedings of the 49th International ACM SIGIR Conference (SIGIR '26), July 20-24 2026, Melbourne"
• source_title: "Pretraining Exposure Explains Popularity Judgments in Large Language Models"
• source_venue: "arXiv preprint 2605.12382 [cs.CL]; accepted SIGIR '26 (ACM), DOI 10.1145/3805712.3809958"
• source_quote: "LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons."
• source_quote_2: "popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias"
• source_url_2: "https://arxiv.org/pdf/2605.12382"
• source_tier: 1
• source_sha: "a03df7824c8b7f91f2b1b5728afacbef8155c15d01bdd7143c32c83a5fe5d807"
• claim type: technical-mechanism (direct empirical test) — floor cleared (Tier 1, accepted ACM SIGIR '26)

Claim: no source splices Simkin & Roychowdhury's copying mechanism to GPT-4's citation bias — the "laundered inheritance" reading is plausible but untested, and the resemblance's status (shared mechanism vs. convergent curve) is unresolved

Targeted search (web search across several query formulations, plus direct inspection of the Algaba, Naser, and Mozafari primaries) found no paper that tests, or even explicitly proposes, the specific two-step chain the topic question names: (1) that the human citation ecosystem's own skew toward highly-cited papers is produced by reference-list copying in Simkin & Roychowdhury's sense — rather than, say, independent accumulating merit-recognition — and (2) that this specific copying-produced skew, rather than some other route to citation-count inflation (media attention, open-access availability, institutional prestige), is what GPT-4 absorbed into its training corpus and is now reproducing. Simkin & Roychowdhury's paper itself predates LLM citation behavior by two decades and makes no claim about machine training; Algaba et al.'s paper, which does study GPT-4's citation behavior, proposes training-data frequency as "one plausible hypothesis" without citing Simkin, Roychowdhury, or any copying-based account of why the training-data frequency distribution looks the way it does.

The existing vault bridge note on the adjacent cosine-0.84 pairing (observation-alemohammad-2026-simkin-2005-cosine-084-pairing-is-analytical-contrast) already establishes that a superficially similar embedding-flagged pairing — Simkin's copying model against Alemohammad et al.'s LLM-dilution benchmark — turned out to be mechanism-level opposites sharing only a higher-order "recognition-free concentration" resonance, not a shared causal pathway. This capture's pairing (Algaba's GPT-4 finding against Simkin's copying model) has not received the same adjudication, and the honest result of this search is that it cannot yet be adjudicated either way: no primary source confirms the inheritance reading (that GPT-4's bias IS Simkin's copying mechanism wearing a different interface), and none explicitly rules it out as mere curve-convergence. The two processes may well turn out, on closer mechanistic inspection, to be a true hand-off (Simkin's copying produces the skewed citation counts; those citation counts determine training-corpus frequency per Algaba's hypothesis and Mozafari et al.'s confirmed general mechanism; frequency determines GPT-4's output preference) — but no single source states or tests this full chain, and the chain's first link (does copying specifically, rather than citation-count inflation generally, explain the training-corpus distribution) has not been examined by anyone.

Central question: [unverified — could not confirm or deny after search]. What is missing, specifically: a study that (a) takes a citation ecosystem whose skew is independently shown (e.g., via misprint-fingerprinting, as in Simkin & Roychowdhury's own 2002 method) to be copying-driven rather than merit-driven, and (b) traces that same ecosystem's text into an LLM's training corpus and (c) shows the LLM's output bias tracks the copying-attributable portion of the skew specifically, not citation count in general. No such study was found.

Further leads

• Niimi (2025), "Hallucinations in Bibliographic Recommendation: Citation Frequency as a Proxy for Training Data Redundancy" (arXiv:2510.25378, GPT-4.1, n=100 generated bibliographic records) — found citation count strongly correlated with factual-recall accuracy (r=0.75, log-linear) and near-verbatim memorization above ~1,000 citations; about hallucination/accuracy, not citation selection bias, but reinforces "citation count tracks training-data redundancy" as a general GPT mechanism. Read in full this session (sha256 aa91ce87a073d1604321f43efa49bd81f6220ef100ae8f2225a31ee5afd5bb2e) but not developed as a core claim since it doesn't test selection bias.
• Naser (2026), ten-LLM citation audit (arXiv:2603.03299, already in the vault) — a WebSearch summary (not independently verified by direct quote this session) suggests the paper also proposes that open-access availability of highly-cited work in training data may partly explain the bias; worth a direct read of the paper's discussion section to check and, if confirmed, add as a second named mechanism-hypothesis distinct from Algaba's.
• Niimi, "How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?" (arXiv:2504.02767, 274,951 GPT-4o-generated references across 10,000 papers) — a larger-N replication of Algaba's citation-popularity finding, surfaced via WebSearch but not fetched or quote-checked this session.
• CiteAgent / "Leveraging LLM-based agents for social science research: insights from citation network simulations" (arXiv:2511.03758) — an LLM-agent framework that reproduces heavy-tailed citation-degree distributions in simulation; surfaced via WebSearch, not fetched. Possibly a more direct test bed for "does an LLM-driven citation process reproduce copying-model statistics," worth a dedicated follow-up.
• Papachristou & Yuan (arXiv:2402.10659) — reports LLM agents (incl. GPT-4o-mini) show preferential attachment in synthetic social-network formation but prioritize triadic closure/homophily in real-world network replication; surfaced via WebSearch, not fetched; a possible analog outside citations for "does the LLM's preference match the generative mechanism, or just the resulting statistic."

Entity candidates

• M.V. Simkin — person — the foundational figure the whole comparison rests on; his and Roychowdhury's 2005 copying model is the older, named mechanism the newer GPT-4 finding is being measured against. Already has an entity page.
• Andres Algaba — person — lead author of the GPT-4 citation-bias paper and source of the "training-data-frequency hypothesis" claim central to this capture.
• Vincent Ginis — person — senior/corresponding-institution author on the Algaba et al. paper (Harvard); co-proposer of the training-frequency hypothesis.
• Jamshid Mozafari — person — lead author of the 2026 pretraining-exposure paper; proposes and tests the general "inheritance" mechanism this capture's core synthesis leans on most heavily.
• Adam Jatowt — person — senior author on the Mozafari et al. paper, University of Innsbruck.
• OLMo / Dolma — concept/term — the fully-open model and pretraining corpus that made Mozafari et al.'s direct exposure measurement possible; worth its own note as the enabling condition for a whole emerging genre of "test the training-data-exposure hypothesis directly" papers, since closed-weight models like GPT-4 cannot be tested this way.
• M.Z. Naser — person — author of the ten-LLM citation audit already in the vault; possibly carries a second, distinct mechanism-hypothesis (open-access availability) not yet checked directly.