LLM errors are correlated well above chance, and the correlation rises, not falls, as paired models get individually more accurate

ai-safetycorrelated-errorsalgorithmic-monoculturellm-evaluationnlp-evaluation

A study measuring error agreement between 349 models on a HuggingFace leaderboard, 71 models on Stanford's HELM benchmark, and 20 models on a resume-screening task finds, for every model pair, how often their wrong answers coincide (conditional on both being wrong). Random disagreement on HELM's typical multiple-choice format would predict roughly 1/3 agreement; the measured mean was 60% on HELM and 42.3% on HuggingFace, with 97.5% and 100% of pairs respectively exceeding the random baseline. A regression shows shared developer, shared base architecture, and similar model size each predict higher agreement — but even controlling for those factors, pairs of models that are each individually more accurate still show higher error correlation: newer models that look different on the surface are converging in their mistakes, not diverging. This sits inside the broader "algorithmic monoculture" framework (entity-algorithmic-monoculture) and extends entity-kenny-peng and entity-nikhil-garg's prior matching-market monoculture work into LLM error correlation directly.