A 2023 study found ChatGPT selects citations by relying solely on Google Scholar citation counts, framed by its author as the Matthew effect operating inside an LLM

chatgptllmcitation-metricsmatthew-effectbibliometricsgoogle-scholarrobert-merton

Eduard Petiška (Charles University, Prague) had GPT-4 write literature-review introductions across ten environmental-science subdisciplines and analyzed the 250 references GPT-4 selected, in "ChatGPT cites the most-cited articles and journals, relying solely on Google Scholar's citation counts. As a result, AI may amplify the Matthew Effect in environmental science" (arXiv 2304.06794, 2023-04-11). Directly, from the paper: "GPT seems to exclusively rely on citation count data from Google Scholar for the works it cites," and, in the paper's own framing of the result, "This finding reinforces the dominance of Google Scholar among scientific databases and perpetuates the Matthew Effect in science, where the rich get richer in terms of citations." The median citation count among the 250 selected references was 1184.5 — a figure that itself shows the selection skewing toward already-famous, older work rather than the most relevant or most recent.

The paper is explicit that this is Robert Merton's 1968 concept operating mechanically inside a large language model: an aggregate citation count, with no separate judgment of a source's actual reliability or relevance to the query, standing in for quality. Merton's own 1968 paper named the underlying pattern in human scientific credit allocation; this is empirical evidence of an AI system reproducing that same pattern in citation selection, fifty-five years later and in an entirely different substrate.

The study is a single-author, non-peer-reviewed preprint — ChatGPT itself is credited in the paper's byline as "Assistant and respondent" — a first look rather than a confirmed finding; a peer-reviewed or larger-N replication would strengthen it considerably. See observation-petiska-chatgpt-matthew-effect-bridges-garfield-warning-and-rag-reliability for how this specific finding connects two previously unlinked vault clusters.