Does LLM citation-popularity bias measurably compound across successive training generations as LLM-selected citations re-enter training corpora?
Raised while promoting observation-petiska-matthew-effect-finding-independently-replicated-by-algaba-and-naser. Petiška's, Algaba et al.'s, and Naser's studies together establish, cross-sectionally, that current LLMs select citations by popularity rather than relevance. None of the three runs a longitudinal or multi-generation experiment: no study located measures whether that specific bias gets worse across successive model generations as LLM-selected, popularity-skewed citations re-enter training corpora that future models are trained on.
The closest evidence found is qualitative and single-instance: Ansari 2026's "Contamination Inheritance" traces one fabricated citation from an earlier paper's text into a later model's output — a real, named mechanism for output-to-training-data propagation, but for citation fabrication, not citation-count popularity, and observed once, not measured at scale. Naser 2026 raises the concern explicitly but does not test it: "As LLMs become more integrated into literature review workflows, this training-data-mediated bias could compound existing disparities in citation patterns."
What would resolve this:
- A multi-generation iterated-retraining or iterated-learning experiment that specifically tracks citation-count popularity skew (not general output quality or "model collapse") across successive training rounds where each round's training corpus includes the prior round's LLM-selected citations.
- Absent a dedicated experiment, a longitudinal audit comparing citation-popularity skew in a fixed LLM family across release generations (e.g. GPT-4 → GPT-4o → GPT-5) trained on progressively more LLM-contaminated web/training data, holding task and domain fixed.
Not the same question as general model-collapse/iterated-learning
degradation, which the vault already covers in depth
(50-questions/_answered/question-verify-schaeffer-2025-eight-model-collapse-definitions.md,
question-verify-nature-2024-iterated-learning-model-collapse-comment.md) —
this question is specifically about citation-count popularity bias, a
narrower mechanism the general literature does not directly test.
Progress log
- 2026-09-14: Still open in its literal form (no located study retrains successive model generations on a corpus containing prior LLM-selected citations and measures whether popularity bias itself strengthens). New adjacent evidence: a twelve-round recursive citation-selection benchmark (fixed models, recycled candidate pool, not retraining) finds that citation concentration on real papers intensifies anyway — through dilution of the authentic-candidate pool, not through the underlying preference growing stronger — and that the cross-vendor "monoculture" behind the original bias collapses once judged candidates are AI-generated. (claim-alemohammad-2026-recursive-citation-benchmark-dilution-concentrates-attention, claim-alemohammad-2026-cross-vendor-citation-monoculture-collapses-under-recursion) This narrows what "compounding" could look like even absent a literal retraining experiment, but does not answer the question as posed.
- 2026-09-14: A second search this session, targeted specifically at the literal retraining-generation mechanism, again found no study that trains a model, has it select citations, feeds those citations into a successor model's training corpus, and measures whether citation-count popularity skew compounds. It did surface the closest existing analog to the requested design: Wang et al. 2024/2025 ran a genuine ten-generation iterated-fine-tuning chain (GPT-2, following the Shumailov et al. model-collapse design) that isolates a specific bias's compounding trajectory from general model collapse — "bias amplification persists independently of model collapse" — but for political-ideology lean in news text, not citation popularity. No citation or bibliometric variable appears anywhere in that design. This shows the requested experimental design is buildable and has already worked for a structurally similar bias; it remains unapplied to citation popularity specifically. Question stays open — partially informed, not answered.