---
title: "Does LLM citation-popularity bias measurably compound across successive training generations as LLM-selected citations re-enter training corpora?"
type: "question"
status: "open"
date_raised: "2026-09-13T00:00:00.000Z"
tags: ["chatgpt","llm","citation-metrics","matthew-effect","model-collapse","feedback-loop","training-data-contamination"]
---


Raised while promoting
[[observation-petiska-matthew-effect-finding-independently-replicated-by-algaba-and-naser]].
[[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|Petiška's]],
[[claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect|Algaba
et al.'s]], and
[[claim-naser-2026-ten-llm-audit-confirms-citation-popularity-bias-across-vendors|Naser's]]
studies together establish, cross-sectionally, that current LLMs select
citations by popularity rather than relevance. None of the three runs a
longitudinal or multi-generation experiment: no study located measures
whether that specific bias gets *worse* across successive model generations
as LLM-selected, popularity-skewed citations re-enter training corpora that
future models are trained on.

The closest evidence found is qualitative and single-instance:
[[claim-ansari-2026-contamination-inheritance-citation-error-propagates-across-models|Ansari
2026's "Contamination Inheritance"]] traces one fabricated citation from an
earlier paper's text into a later model's output — a real, named mechanism
for output-to-training-data propagation, but for citation *fabrication*, not
citation-count *popularity*, and observed once, not measured at scale.
Naser 2026 raises the concern explicitly but does not test it: "As LLMs
become more integrated into literature review workflows, this
training-data-mediated bias could compound existing disparities in citation
patterns."

What would resolve this:

1. A multi-generation iterated-retraining or iterated-learning experiment
   that specifically tracks citation-count popularity skew (not general
   output quality or "model collapse") across successive training rounds
   where each round's training corpus includes the prior round's
   LLM-selected citations.
2. Absent a dedicated experiment, a longitudinal audit comparing
   citation-popularity skew in a fixed LLM family across release generations
   (e.g. GPT-4 → GPT-4o → GPT-5) trained on progressively more
   LLM-contaminated web/training data, holding task and domain fixed.

Not the same question as general model-collapse/iterated-learning
degradation, which the vault already covers in depth
(`50-questions/_answered/question-verify-schaeffer-2025-eight-model-collapse-definitions.md`,
`question-verify-nature-2024-iterated-learning-model-collapse-comment.md`) —
this question is specifically about citation-count popularity bias, a
narrower mechanism the general literature does not directly test.

## Progress log

- 2026-09-14: Still open in its literal form (no located study retrains
  successive model *generations* on a corpus containing prior LLM-selected
  citations and measures whether popularity bias itself strengthens). New
  adjacent evidence: a twelve-round recursive citation-*selection*
  benchmark (fixed models, recycled candidate pool, not retraining) finds
  that citation concentration on real papers intensifies anyway — through
  dilution of the authentic-candidate pool, not through the underlying
  preference growing stronger — and that the cross-vendor "monoculture"
  behind the original bias collapses once judged candidates are
  AI-generated.
  ([[claim-alemohammad-2026-recursive-citation-benchmark-dilution-concentrates-attention]],
  [[claim-alemohammad-2026-cross-vendor-citation-monoculture-collapses-under-recursion]])
  This narrows what "compounding" could look like even absent a literal
  retraining experiment, but does not answer the question as posed.
- 2026-09-14: A second search this session, targeted specifically at the
  literal retraining-generation mechanism, again found no study that trains
  a model, has it select citations, feeds those citations into a
  successor model's training corpus, and measures whether citation-count
  popularity skew compounds. It did surface the closest existing analog to
  the requested design:
  [[claim-wang-2024-bias-amplification-persists-independent-of-model-collapse|Wang
  et al. 2024/2025]] ran a genuine ten-generation iterated-fine-tuning
  chain (GPT-2, following the Shumailov et al. model-collapse design) that
  isolates a *specific* bias's compounding trajectory from general model
  collapse — "bias amplification persists independently of model
  collapse" — but for political-ideology lean in news text, not citation
  popularity. No citation or bibliometric variable appears anywhere in
  that design. This shows the requested experimental design is buildable
  and has already worked for a structurally similar bias; it remains
  unapplied to citation popularity specifically. Question stays open —
  partially informed, not answered.