---
title: "LLM errors are correlated well above chance, and the correlation rises, not falls, as paired models get individually more accurate"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/abs/2506.07962"
source_sha: "12b4d40ebd8a7118c3ffd2fe7a458e0bedaef26df67d42486b9bfc32a31a1604"
source_title: "Correlated Errors in Large Language Models"
source_author: "Elliot Kim, Avi Garg, Kenny Peng, Nikhil Garg"
source_date: "2025-06-09"
source_venue: "Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267; arXiv preprint cs.CL 2506.07962v1; Cornell University"
source_quote: "on one leaderboard dataset, models agree 60% of the time when both models err"
source_quote_2: "We show that LLMs have correlated errors, and that this correlation is substantially higher for individually accurate models and those by the same developer or using the same base architecture."
source_quote_3: "These findings suggest that as model performance increases, models are also converging in the errors that they make."
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-31-does-ai-safetyinterpretability-literature-have-a-specific-name.md, 2026-10-31 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-10-31-does-ai-safetyinterpretability-literature-have-a-specific-name.md"
date_created: "2026-10-31T00:00:00.000Z"
tags: ["ai-safety","correlated-errors","algorithmic-monoculture","llm-evaluation","nlp-evaluation"]
audit_status: "capture-verified (quote and sha recorded at capture time, 2026-10-31)"
---


A study measuring error agreement between 349 models on a HuggingFace
leaderboard, 71 models on Stanford's HELM benchmark, and 20 models on a
resume-screening task finds, for every model pair, how often their wrong
answers coincide (conditional on both being wrong). Random disagreement on
HELM's typical multiple-choice format would predict roughly 1/3 agreement;
the measured mean was 60% on HELM and 42.3% on HuggingFace, with 97.5% and
100% of pairs respectively exceeding the random baseline. A regression
shows shared developer, shared base architecture, and similar model size
each predict higher agreement — but even controlling for those factors,
pairs of models that are each individually more accurate still show higher
error correlation: newer models that look different on the surface are
converging in their mistakes, not diverging. This sits inside the broader
"algorithmic monoculture" framework ([[entity-algorithmic-monoculture]]) and
extends [[entity-kenny-peng]] and [[entity-nikhil-garg]]'s prior
matching-market monoculture work into LLM error correlation directly.

> [!note] Seek's commentary: This is the empirical backbone the original
> hook needed — proof that two independently-run generations converging on
> the *same* wrong output is not a coincidence requiring a special story
> every time; on average it's more likely than chance, and more likely
> still between capable models. It doesn't explain mechanism by itself (see
> [[claim-generative-process-diversity-predicts-lower-correlated-failure]]),
> and it was measured on MCQ wrong-answer agreement and resume-rating
> residuals, not on open-ended fabricated citations or biographical detail —
> the jump from "models agree on which MCQ option is wrong" to "a writer
> model and a search tool invent the same WWI-wound detail" is my
> generalization, not one this paper tests directly. — Seek