Divergence, not convergence, is the literature's own default outcome for an open-ended fabricated detail with no strong training-data regularity behind it

ai-safetyhallucinationfabricationcorrelated-errorsnlp-evaluation

OpenAI's account of why language models hallucinate frames the failure as a byproduct of a binary classification problem implicitly solved during training (distinguishing "valid" continuations from errors), worsened by evaluation procedures that reward confident guessing over admitting uncertainty. The paper draws a plain counter-example to convergence: asked for the title of a specific person's PhD dissertation, three different widely-used models each returned a different fabricated title, school, and year — "None generated the correct title or year" — and asked to count a specific letter in a word, DeepSeek-V3 "returned '2' or '3' in ten independent trials," with other models answering as high as 6 or 7. The paper's own framing treats hallucination on "arbitrary" facts — those with no learnable pattern in training data — as statistically expected, distinct from facts governed by a learnable regularity, and this distinction is the theoretical descendant of the same authors' earlier "Calibrated Language Models Must Hallucinate" (2023/STOC 2024). This directly complicates claim-correlated-errors-in-llms-exceed-chance-and-rise-with-accuracy: that paper shows correlation above chance on average across model pairs, while this one shows a documented case of three models scattering to three different wrong answers on one open-ended question — convergence is not automatic even where correlated error is, on average, the rule.