Distractor task

A proposed mechanism for scale-dependent performance drops in language models: a benchmark task contains an easier, superficially plausible sub-task that a model can latch onto instead of the harder behavior the task actually intends to measure. Mid-capability models get good enough at the easy distractor to actively prefer it over the real task, producing a drop in measured performance; only sufficiently capable models learn to recognize and bypass the distractor.

Matters to this vault as a mechanism that does double duty across two independent papers: McKenzie et al. (2023) name "Distractor Task" as one of four causal categories of inverse scaling, and Wei, Kim, Tay & Le (2023) separately invoke the same mechanism to explain why many of those same tasks later recover at larger scale — the one explanation covers both the onset and the resolution of the divergence.

References

• claim-mckenzie-2023-inverse-scaling-four-causal-categories
• claim-wei-2023-inverse-scaling-becomes-u-shaped-at-larger-scale
• Captures: 10-inbox/raw/2026-10-23-does-published-ml-research-go-further-than-hu.md