Mozafari et al. 2026 find LLM popularity judgments track pretraining-exposure frequency more closely than real-world popularity signals like Wikipedia pageviews

llm-citation-biaspretraining-exposurepopularity-biasmatthew-effectolmodolmatraining-data-frequency

Mozafari, Piryani & Jatowt (University of Innsbruck) built the first large-scale direct test of whether an LLM's sense of how "popular" something is tracks real-world popularity or merely the statistical frequency of that thing's appearance during pretraining. Using the fully open OLMo model family and its complete Dolma pretraining corpus (7.4 trillion tokens, indexed with the Infini-gram toolkit), they computed exact entity-level exposure counts for 2,000 entities across five types — Person, Location, Organization, Art, Product — something impossible on a closed-weight model like GPT-4, whose training data is not published. They compared this ground-truth exposure measure against Wikipedia pageviews (an external, independent popularity signal) and two elicited LLM popularity judgments (direct scalar estimation and pairwise comparison).

Their finding: "LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons," an effect strongest for larger models and persisting "in the long tail, where Wikipedia popularity becomes unreliable." Their own conclusion states the mechanism plainly: "popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias."

The paper does not address citation selection and does not cite Algaba et al., Petiška, or any bibliometric copying model — its entities are people, places, organizations, artworks, and products, not papers or citation counts. It is nonetheless the closest available direct, controlled confirmation of the general mechanism-class that Algaba et al.'s own Discussion only proposes as a hypothesis for their GPT-4 citation finding: that an LLM's preference is an inherited property of training-corpus frequency rather than an independently formed judgment of real-world merit or fame. See question-does-citation-copying-explain-training-corpus-skew-behind-llm-citation-bias for what would be needed to extend this confirmed general mechanism to citations specifically.