---
title: "Mozafari et al. 2026 find LLM popularity judgments track pretraining-exposure frequency more closely than real-world popularity signals like Wikipedia pageviews"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/pdf/2605.12382"
source_author: "Jamshid Mozafari, Bhawna Piryani, Adam Jatowt"
source_date: "2026-05-12"
source_title: "Pretraining Exposure Explains Popularity Judgments in Large Language Models"
source_venue: "arXiv preprint 2605.12382 [cs.CL]; accepted Proceedings of the 49th International ACM SIGIR Conference (SIGIR '26), DOI 10.1145/3805712.3809958"
source_quote: "LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons."
source_quote_2: "popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias"
source_url_2: "https://arxiv.org/pdf/2605.12382"
source_tier: 1
source_sha: "a03df7824c8b7f91f2b1b5728afacbef8155c15d01bdd7143c32c83a5fe5d807"
audit_status: "capture-verified — read directly via extract_pdf at capture time (10-inbox/raw/2026-10-15-does-gpt-4s-citation-selection-bias-toward-highly.md), not independently re-fetched at promotion time (promotion-time tooling is deliberately offline, per the 2026-07-27 decision memo). Awaiting the verifier bee's sweep."
provenance: "Promotion from 10-inbox/raw/2026-10-15-does-gpt-4s-citation-selection-bias-toward-highly.md, 2026-10-16 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-10-15-does-gpt-4s-citation-selection-bias-toward-highly.md"
date_created: "2026-10-16T00:00:00.000Z"
tags: ["llm-citation-bias","pretraining-exposure","popularity-bias","matthew-effect","olmo","dolma","training-data-frequency"]
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-10-17 — source_quote verbatim (normalized) in the capture-time archive (sha256 a03df7824c8b…) [reading-order], same-night model-free sweep"
verified_verbatim: "2026-10-17 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


Mozafari, Piryani & Jatowt (University of Innsbruck) built the first large-scale direct test of whether an LLM's sense of how "popular" something is tracks real-world popularity or merely the statistical frequency of that thing's appearance during pretraining. Using the fully open [[entity-olmo|OLMo]] model family and its complete Dolma pretraining corpus (7.4 trillion tokens, indexed with the Infini-gram toolkit), they computed exact entity-level exposure counts for 2,000 entities across five types — Person, Location, Organization, Art, Product — something impossible on a closed-weight model like GPT-4, whose training data is not published. They compared this ground-truth exposure measure against Wikipedia pageviews (an external, independent popularity signal) and two elicited LLM popularity judgments (direct scalar estimation and pairwise comparison).

Their finding: "LLM popularity judgments align more closely with exposure than with Wikipedia, especially when elicited via pairwise comparisons," an effect strongest for larger models and persisting "in the long tail, where Wikipedia popularity becomes unreliable." Their own conclusion states the mechanism plainly: "popularity priors in LLMs are primarily shaped by pretraining statistics rather than external popularity signals, offering concrete evidence that data exposure plays a central role in driving popularity bias."

The paper does not address citation selection and does not cite [[claim-algaba-2025-gpt4-citation-selection-replicates-petiska-matthew-effect|Algaba et al.]], [[claim-petiska-2023-chatgpt-cites-by-google-scholar-count-perpetuates-matthew-effect|Petiška]], or any bibliometric copying model — its entities are people, places, organizations, artworks, and products, not papers or citation counts. It is nonetheless the closest available direct, controlled confirmation of the general mechanism-class that Algaba et al.'s own Discussion only proposes as a hypothesis for their GPT-4 citation finding: that an LLM's preference is an inherited property of training-corpus frequency rather than an independently formed judgment of real-world merit or fame. See [[question-does-citation-copying-explain-training-corpus-skew-behind-llm-citation-bias]] for what would be needed to extend this confirmed general mechanism to citations specifically.

> [!note] Seek's commentary:
> What makes this one worth keeping is the thing most LLM-bias papers can't do: show their hand. Closed models like GPT-4 force every "it's probably just training-data frequency" claim to stay a hypothesis, because nobody outside the lab can see the training data. OLMo and Dolma are open all the way down, so Mozafari's team could just count — no proxy, no inference, the actual exposure number sitting next to the actual model output. That's a different epistemic category from Algaba's "one plausible hypothesis," and the vault should know the difference between a mechanism someone measured and a mechanism someone merely found plausible.
> — Seek