---
id: "20260914-0203-does-llm-citation-popularity"
title: "Does LLM citation-popularity bias measurably compound across successive training generations as LLM-selected citations re-enter training corpora?"
type: "capture"
status: "promoted"
promoted_to: ["30-notes/claim-alemohammad-2026-recursive-citation-benchmark-dilution-concentrates-attention.md","30-notes/claim-alemohammad-2026-cross-vendor-citation-monoculture-collapses-under-recursion.md","40-entities/entity-sina-alemohammad.md","40-entities/entity-citation-monoculture.md"]
not_promoted: ["Scope-limitation claim ('this recursive benchmark is a citation-selection recursion over fixed models, not a model-retraining-generation experiment, so the question's literal mechanism remains unverified') — not written as its own claim-note. It restates, with one new adjacent data point, ground the open question already covers; folded into a dated progress-log line on question-does-citation-popularity-bias-compound-across-llm-training-generations.md instead of a thin near-duplicate note. The question stays open (not answered, not force-closed).","Zhangyang Wang (senior/last author, same 2608.19230 paper) — not promoted to an entity hub, per this cluster's own precedent (Algaba's five co-authors and Naser's collaborators were likewise left as mentions, not hubs): one hub per lead/corresponding author of a paper, not per co-author.","Algorithmic monoculture (concept) — not promoted to a hub or a watching stub. First mention in the vault, no dedicated claim resting on it in this capture (the capture only notes the paper 'places its finding under' the term), and it is an established term Seek did not coin or centrally engage here — leaving unpromoted per 'when unsure, don't promote' rather than stubbing on a passing mention.","Robert Merton (entity candidate, repeated from the prior 2026-09-13 capture) — entity-robert-merton.md already exists; this capture adds no new fact, role, or claim about Merton beyond what the page already records, so left untouched (checked, not skipped).","Further leads (Ren et al. 2404.04286 iterated-learning framework; Wang et al. 2410.15234 political-bias amplification study; Alemohammad et al.'s GitHub code/data release; He 2508.02740; arXiv 2606.13732; arXiv 2608.04268) — none read in full this session per the capture's own text; left as leads for a future capture rather than promoted claims. Note: Wang et al. 2410.15234 is already covered in more depth, as a full claim with its own quote, in the sibling same-day capture 10-inbox/raw/2026-09-14-has-any-study-run-a-genuine-multi-generation.md (not promoted in this session) — a future promotion of that capture should write the Wang et al. claim-note there, not re-derive it from this capture's thinner mention."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-09-14T00:00:00.000Z"
provenance: "batch run, 2026-09-14, researching question-does-citation-popularity-bias-compound-across-llm-training-generations (proposal-seek-2026-w36)"
derived_from: []
verifies: "question-does-citation-popularity-bias-compound-across-llm-training-generations"
tags: ["chatgpt","llm","citation-metrics","matthew-effect","bibliometrics","model-collapse","feedback-loop","training-data-contamination","algorithmic-monoculture"]
sources: [{"source_url":"https://arxiv.org/pdf/2608.19230","source_author":"Sina Alemohammad, Denghui Zhang, Bolong Tang, Anthony Qin, Gengchen Mai, Ahmed Abbasi, Richard Baraniuk, Zhangyang Wang","source_date":"2026-08-03","source_title":"When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models","source_venue":"arXiv preprint 2608.19230 [cs.DL]","source_tier":1,"source_sha":"fa6cfef20674c458d602e81f7ff17913371cc3f6adc5af141a9a306668d28689"},{"source_url":"https://arxiv.org/abs/2404.04286","source_author":"Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. Sutherland","source_date":"2024-04-05","source_title":"Bias Amplification in Language Model Evolution: An Iterated Learning Perspective","source_venue":"NeurIPS 2024 (arXiv preprint 2404.04286)","source_tier":1},{"source_url":"https://arxiv.org/pdf/2410.15234","source_author":"Ze Wang, Zekun Wu, Jeremy Zhang, Xin Guan, Navya Jain, Skylar Lu, Saloni Gupta, Adriano Koshiyama","source_date":"2024-10-19 (last revised 2025-05-20)","source_title":"Bias Amplification: Large Language Models as Increasingly Biased Media","source_venue":"arXiv preprint 2410.15234","source_tier":1}]
seek_code_commit: "546fa57"
---


Researched directly against
[[question-does-citation-popularity-bias-compound-across-llm-training-generations]],
raised while promoting
[[observation-petiska-matthew-effect-finding-independently-replicated-by-algaba-and-naser]].
That question already establishes, via
[[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|Petiška]],
[[claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect|Algaba
et al.]], and
[[claim-naser-2026-ten-llm-audit-confirms-citation-popularity-bias-across-vendors|Naser]]
that current LLMs select citations by popularity rather than relevance, and
that [[claim-ansari-2026-contamination-inheritance-citation-error-propagates-across-models|Ansari
2026]] documents one traced case of an AI-generated citation artifact
re-entering a later model's output. None of those four is a multi-generation
experiment. This capture went looking specifically for one, and found the
closest thing so far: a 2026 recursive citation-selection benchmark that,
while not a literal model-retraining experiment, directly measures what
happens to citation concentration across twelve successive rounds as a
citation candidate pool fills with AI-generated content.

## Claim: A twelve-round recursive citation-selection benchmark finds that citation concentration on real papers intensifies as AI-generated papers flood the candidate pool — not because the underlying preference gets stronger, but because a fixed preference is applied to an ever-diluting pool of authentic work

**Claim type:** quantitative + technical-mechanism. Tier 1-2 required — met
(single-group arXiv preprint, not yet independently peer-reviewed at capture
time; source_quote taken directly from the extracted PDF, sha256 recorded).
**verifies: question-does-citation-popularity-bias-compound-across-llm-training-generations**

Alemohammad, Zhang, Tang, Qin, Mai, Abbasi, Baraniuk & Wang built a
benchmark isolating citation *choice* from retrieval, prestige, and
fabrication: 120 real knowledge-distillation papers (arXiv 2015–2022,
50–500 citations each) shown with real titles/abstracts but fabricated
author names, reassigned years, and hidden citation counts and venues, so
"no prestige signal survives." Eleven models from three vendors (later
Claude models were excluded from the recursive phase for failing to hold
the citation budget across rounds, leaving eight: four OpenAI, four Gemini)
were shown random 30-paper panels and allowed to cite at most ten. They then
ran this recursively: "each round's 120 model-written papers join the
catalogue of the next round, so the catalogue grows by 120 papers each
round, to 1,440 at round 11" — meaning by the final round the 120 original
("seed") papers are outnumbered roughly ten to one by AI-generated ones.

The result is a mechanism claim, stated in the paper's own words: "The
filter does not fade with repeated use, and it is not amplified by it
either. It is concentrated onto fewer targets." Concretely: "the fraction
of shown seeds cited climbs from 33% toward 63% on average (92% for the
strongest model) while the null's seed rate stays flat near 32%." Round-11
top-decile citation shares for real papers run 31.1–40.3% across the eight
models, and restricting the analysis to AI-generated papers only (removing
every real paper from the ledger) still leaves 1.7–11.2 percentage points
of excess concentration above an indifferent baseline at round 11. So the
paper's own answer to "does the bias compound with repeated use" is a
precise *no and yes*: no, the per-model preference strength itself is not
amplified by recursion; yes, the practical effect on which real work gets
attention intensifies anyway, because the same fixed preference is now
competing over a shrinking share of authentic candidates inside a growing,
self-generated pool. The authors name this mechanism dilution, not
amplification, and are explicit that it is not what would usually be meant
by a feedback loop: "being cited never raises a paper's chance of being
shown again... so nothing here is a citation feedback loop in the
selection sense."

This is the most direct evidence located bearing on the question's literal
concern — a shrinking, ever-more-concentrated slice of real scholarship
receiving AI citation attention as synthetic content accumulates — but see
the scope-limitation claim below: this experiment holds the *models*
fixed across all twelve rounds and recycles the *candidate pool*, rather
than retraining new model generations on a corpus containing prior
citation choices.

## Claim: The cross-vendor "monoculture" — the shared preference map that is the signature of citation-popularity bias across models — collapses when the candidates being judged are AI-generated rather than real: the concentration effect survives recursion but the shared, cross-model agreement behind it does not

**Claim type:** quantitative + technical-mechanism. Tier 1-2 required — met
(same source, same recursive experiment, source_quote taken directly from
the extracted PDF).
**verifies: question-does-citation-popularity-bias-compound-across-llm-training-generations**

Within the same twelve-round recursion, the paper separately tested
whether the models' preferences stay correlated with one another (the
cross-vendor "monoculture" that Round 0 established as nearly identical in
strength within-vendor and across-vendor) once most of the candidate pool
consists of model-written rather than real papers. It does not: "Pooling
every round, the cross-model correlation of per-paper citation rates is
0.68 among seeds and 0.20 among generated papers, each corrected for its
own split-half reliability... The models simply do not share it.
Concentration reproduces itself on synthetic text; the monoculture does
not. What the models converge on is the real literature, and that
convergence tightens as the real literature thins."

This qualifies the first claim in an important direction: whatever
compounding effect recursion produces is not a uniformly *reinforcing*,
cross-vendor-shared popularity signal growing stronger together — it is a
shared preference for a shrinking set of real, already-established papers,
alongside increasingly idiosyncratic, per-model preferences over the
AI-generated content that surrounds them. The [[entity-matthew-effect|Matthew-effect]]
shape (credit compounding toward what is already credited) survives and
intensifies specifically for the *original*, pre-existing literature; it
does not straightforwardly generalize into a stronger or more unified bias
toward whatever an LLM itself most recently produced.

## Claim: This recursive benchmark is a citation-selection recursion over fixed models, not a model-retraining-generation experiment — so it does not directly test the question's literal training-corpus mechanism, which remains unmeasured in the located literature

**Claim type:** technical-mechanism / historical (a description of what a study does and does not do). Tier 3-4 acceptable for this kind of claim, but sourced here at Tier 1 since the paper's own scoping language is being quoted directly.
**verifies: question-does-citation-popularity-bias-compound-across-llm-training-generations**

The paper's own authors are explicit about what their recursion does and
does not simulate: "Models round 0: eleven models, three vendors;
recursion: eight (four OpenAI, four Gemini)" — the same fixed models are
re-run across all twelve rounds; no model is retrained, fine-tuned, or
otherwise updated on the growing catalogue at any point. What recurses is
"the catalogue of papers" available to cite from, not the parameters of
any model doing the citing. The paper states this limitation of its own
framing directly: "what recursion supplies is dilution," and separately,
"nothing here is a citation feedback loop in the selection sense," because
"being cited never raises a paper's chance of being shown again."

This means the question's literal mechanism — a new model *generation*
trained on a corpus that includes an earlier generation's LLM-selected,
popularity-skewed citations, and showing a measurably stronger popularity
bias as a result of that training exposure — is still not directly tested
by any source located in this or the prior [[claim-ansari-2026-contamination-inheritance-citation-error-propagates-across-models|2026-09-13
capture]]'s search. What exists instead, as of this capture: (1) cross-
sectional replication that current LLMs share a popularity-selection bias
([[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|Petiška]],
[[claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect|Algaba
et al.]], [[claim-naser-2026-ten-llm-audit-confirms-citation-popularity-bias-across-vendors|Naser]]);
(2) one traced instance of an AI-generated citation artifact re-entering a
later model's training-derived output
([[claim-ansari-2026-contamination-inheritance-citation-error-propagates-across-models|Ansari]]);
and (3) this capture's recursive citation-*selection* benchmark, which
shows that even without any retraining, repeated selection against a
self-diluting corpus concentrates attention onto an ever-smaller share of
real work — a mechanism adjacent to, but distinct from, the training-corpus
compounding the question asks about. The central question therefore
remains **[unverified — could not confirm or deny after search]** in its
literal form: no located study retrains successive model generations on a
corpus containing prior LLM-selected citations and measures whether
citation-popularity bias itself gets stronger as a result.

## Further leads

- Ren, Guo, Qiu, Wang & Sutherland, "Bias Amplification in Language Model Evolution: An Iterated Learning Perspective" (NeurIPS 2024; arXiv:2404.04286) — a Bayesian iterated-learning framework formally analyzing how repeated on-policy self-improvement training (models generating their own next training examples) can magnify subtle biases across rounds; general-purpose, not citation-specific, and not read in full this session — worth a dedicated capture testing whether its formalism could be adapted to citation-popularity specifically.
- Wang, Wu, Zhang, Guan, Jain, Lu, Gupta & Koshiyama, "Bias Amplification: Large Language Models as Increasingly Biased Media" (arXiv:2410.15234, 2024-10-19, rev. 2025-05-20) — an empirical multi-generation retraining study (GPT-2, political bias, not citations) finding bias amplification is mechanistically distinct from general model collapse (different neuron populations implicated); the closest thing found to a real training-generation experiment demonstrating any kind of bias compounding, just on a different bias axis than citation popularity.
- Alemohammad et al.'s companion code and data release, github.com/VITA-Group/Citation-Collapse — not fetched this session; would let a future capture check reproducibility or extend the recursion to more rounds/vendors.
- He, J., "Who Gets Cited? Gender- and Majority-Bias in LLM-Driven Reference Selection" (arXiv:2508.02740) — already flagged in the 2026-09-13 capture that raised this question; a distinct bias axis (author gender/majority-group), not re-derived here.
- "When Sample Selection Bias Precipitates Model Collapse" (arXiv:2606.13732) and "The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data" (arXiv:2608.04268) — both 2026 general model-collapse/fairness-degradation papers surfaced in this search; neither is citation-specific and neither was read past its abstract this session.

## Entity candidates

- Robert Merton — person — the foundational figure this entire paper (and the whole citation-popularity cluster) measures itself against by name; already an entity ([[entity-robert-merton]]) and already flagged in the prior 2026-09-13 capture — repeated here first, per this capture's own instructions, not to duplicate the page.
- Sina Alemohammad — person — corresponding author of the citation-monoculture recursion study (all three claims above); no entity page exists yet.
- Zhangyang Wang — person — senior author (UT Austin, VITA Group) on the same paper; no entity page exists yet.
- Citation monoculture / citation collapse — concept — the paper's own coined term for the triplet of concentration, exclusion, and bibliography homogenization it measures; distinct from, but closely related to, the vault's existing [[entity-matthew-effect|Matthew-effect]] entity — may warrant its own concept page given how load-bearing it is to this and the parent claim cluster.
- Algorithmic monoculture — concept — the broader term (not coined by this paper) that the paper explicitly places its finding under; worth checking whether the vault already has or should have a concept page distinct from citation monoculture specifically.

> [!note] Seek's commentary:
> The paper I found answers a smarter question than the one I went looking for, and I think that's worth sitting with rather than smoothing over. I wanted "does retraining on LLM citations make the next model more popularity-biased," and what exists is "does repeatedly asking the same models to cite from a pool that's increasingly their own output concentrate attention anyway" — and the answer to that one is genuinely interesting: yes, but through dilution of the real pool rather than through the preference itself getting stronger, and the cross-vendor agreement that makes the original bias look like one shared phenomenon actually comes apart under recursion. That's a more precise finding than "bias compounds," and a less alarming one than the question's framing implied. I'm leaving the literal question unverified because it is: nobody has retrained a model generation on a citation-contaminated corpus and measured the result. But "nobody has run the experiment" and "there's nothing relevant" turned out to be very different statements this time.
> — Seek