capture promoted 2026-10-23

Does published ML research go further than Hu et al. 2026 in characterizing which task types plateau or degrade under scale — is there a task-taxonomy for the divergence?

neural-scaling-laws inverse-scaling u-shaped-scaling task-taxonomy downstream-scaling scaling-laws averaging-artifact

Framing note (read before the claims below)

The topic asks whether anyone has done for neural scaling laws what Heathcote et al. (2000) did for the power law of practice: not just show the aggregate hides structure, but characterize the structure itself — a taxonomy of which units behave which way, and why. Hu et al. 2026 only flags the phenomenon ("some improve monotonically, others plateau, and some even degrade with scale") in a single sentence of background before moving to its own predictive method. The search below finds that the taxonomy already exists in the literature Hu et al. 2026 itself cites — and in one case was built by two of its own co-authors a year earlier. The answer is "yes, published research goes further," but the taxonomy is a behavior-and-cause classification across many tasks and many models, not a per-individual-unit curve fit the way Heathcote's was per-learner; that distinction matters and is addressed in the commentary below.

Claim: McKenzie et al. (2023) classify inverse-scaling tasks into four causal categories, not just a single "inverse scaling" label — the first task-type taxonomy for scale-driven degradation

McKenzie, Lyzhov, Pieler, Parrish et al., "Inverse Scaling: When Bigger Isn't Better" (arXiv:2306.09479, published in Transactions on Machine Learning Research, 10/2023), ran a two-round public contest (the Inverse Scaling Prize) that collected 11 tasks on which large-language-model performance gets worse with scale, then went past merely cataloguing them: "Through analysis of the datasets, along with other examples found in the literature, we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task." The paper labels these causes "Strong Prior," "Unwanted Imitation," "Distractor Task," and "Spurious Few-Shot" respectively, and assigns each of the 11 prize-winning tasks to one of the four categories (e.g. Memo Trap and Prompt Injection under Strong Prior; Modus Tollens under Unwanted Imitation; Pattern Match Suppression and NeQA under Distractor Task; Hindsight Neglect and Repetitive Algebra under Spurious Few-Shot).

This is a mechanism-level taxonomy — not "this task plateaus" but "this task plateaus because the model prefers a memorized completion over an instructed one" — of exactly the kind the topic asks whether anyone has built. It predates Hu et al. 2026 by three years and is the paper Hu et al. 2026 itself cites for "a phenomenon known as inverse scaling."

source_url: https://arxiv.org/abs/2306.09479 source_sha: 825e7953b35eb5676a40bf4552f169fbccc542eeec0b680860da2ae15a7ad9d7 source_title: "Inverse Scaling: When Bigger Isn't Better" source_author: "Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, et al. (Inverse Scaling Prize winning-task authors), Najoung Kim, Samuel R. Bowman, Ethan Perez" source_date: 2023-10 source_venue: "Transactions on Machine Learning Research (TMLR); arXiv:2306.09479" source_quote: "we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task." source_tier: 1 source_delight: "The paper names and justifies each of its four causal categories with a worked example task, so the taxonomy can be checked against the original prompts (e.g. the π-redefinition task under 'Strong Prior') rather than taken on the authors' label alone."

Claim: Wei, Kim, Tay & Le (2023) show the inverse/degrade label is scale-dependent, not a fixed property of a task — at larger scale, most of McKenzie et al.'s "inverse scaling" tasks reverse into "U-shaped scaling" instead

Wei, Kim, Tay & Le, "Inverse scaling can become U-shaped" (arXiv:2211.02011, published EMNLP 2023), re-ran McKenzie et al.'s 11 inverse-scaling tasks on PaLM models up to 540B parameters — roughly five times the training compute of the models evaluated in the original Inverse Scaling Prize. "With this increased range of model sizes and training compute, only four out of the eleven tasks remain inverse scaling. Six out of the eleven tasks exhibit "U-shaped scaling"," meaning performance first drops with scale and then recovers at the largest sizes tested, rather than continuing to degrade. The authors propose a mechanism for the reversal: "which we attribute to the presence of distractor tasks that only sufficiently large models can avoid" — i.e., mid-sized models get good enough at a tempting-but-wrong sub-task to actively hurt performance, while only the largest models learn to bypass the distractor and do the intended task.

This complicates any static task-taxonomy: a task's category (plateau, degrade, U-shaped, monotonic improvement) is not fixed to the task alone but depends on where in the scale range it is measured — the same task can read as "inverse scaling" at one scale and "U-shaped" or "positive" at another. Any taxonomy of which task types diverge under scale therefore needs a scale-position axis, not just a task-identity axis.

source_url: https://arxiv.org/abs/2211.02011 source_sha: 2b5c548e9ec118a669215fbca7b5803fc4a0900d5658913cc66cbd95eb054449 source_title: "Inverse scaling can become U-shaped" source_author: "Jason Wei, Najoung Kim, Yi Tay, Quoc V. Le" source_date: 2023 source_venue: "Google; arXiv:2211.02011, published as EMNLP 2023 (aclanthology 2023.emnlp-main.963)" source_quote: "With this increased range of model sizes and training compute, only four out of the eleven tasks remain inverse scaling. Six out of the eleven tasks exhibit “U-shaped scaling”" source_quote_2: "which we attribute to the presence of distractor tasks that only sufficiently large models can avoid" source_url_2: https://arxiv.org/abs/2211.02011 source_sha_2: 2b5c548e9ec118a669215fbca7b5803fc4a0900d5658913cc66cbd95eb054449 source_tier: 1

Claim: Lourie, Hu & Cho (2025) — co-authored by two of Hu et al. 2026's own authors a year earlier — build an explicit six-category taxonomy of downstream scaling behaviors and quantify how rare the "clean power law" case actually is

Lourie, Hu & Cho, "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check" (arXiv:2507.00885, published in Findings of EMNLP 2025), re-examined the 46 downstream tasks studied by Gadre et al. (2025) and report: "we re-examine scaling behavior on the 46 tasks tested by Gadre et al. (2025), classifying them into six categories visualized in Figure 2. We find that linear scaling actually occurs in a minority of cases in their setting: 39% of the time." Their Figure 2 is captioned explicitly as "A taxonomy of different scaling behaviors. Predictable scaling fits closely to a linear functional form after, for example, exponentiating the cross-entropy loss. However, depending on the downstream task, models do not always improve with scale (inverse, nonmonotonic, and trendless), or the improvement might be highly noisy." (A sixth category, "breakthrough" — a sigmoid-shaped jump — rounds out the six.) Of the 46 tasks, only 18 (39%) showed the smooth, predictable improvement that downstream scaling laws assume; the other 28 were distributed across the five degenerate categories (inverse, nonmonotonic, noisy, trendless, breakthrough).

Two of this paper's three authors (Michael Y. Hu and Kyunghyun Cho) are co-authors of Hu et al. 2026, and Hu et al. 2026 cites this paper directly for the claim that "no single parametric family can capture the full spectrum of scaling behaviors [Lourie et al., 2025]." This is the most direct answer available to the topic question: a task-taxonomy for the scaling divergence exists, is quantified across real tasks, and was published by the same research group one year before Hu et al. 2026 — which cites it as settled background rather than extending it.

source_url: https://arxiv.org/abs/2507.00885 source_sha: c88965b755cab41dd7d42063a587ee4c47532e4fc5c8b4d1d2c35405bc7bb1fa source_title: "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check" source_author: "Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho" source_date: 2025 source_venue: "New York University; Findings of the Association for Computational Linguistics: EMNLP 2025; arXiv:2507.00885" source_quote: "we re-examine scaling behavior on the 46 tasks tested by Gadre et al. (2025), classifying them into six categories visualized in Figure 2. We find that linear scaling actually occurs in a minority of cases in their setting: 39% of the time" source_quote_2: "A taxonomy of different scaling behaviors. Predictable scaling fits closely to a linear functional form after, for example, exponentiating the cross-entropy loss. However, depending on the downstream task, models do not always improve with scale (inverse, nonmonotonic, and trendless), or the improvement might be highly noisy." source_url_2: https://arxiv.org/abs/2507.00885 source_sha_2: c88965b755cab41dd7d42063a587ee4c47532e4fc5c8b4d1d2c35405bc7bb1fa source_tier: 1 source_delight: "Figure 2's caption literally uses the word 'taxonomy' for exactly the thing this topic asks whether anyone has built — and the authors show in a follow-on section that the same task's category can flip between two nominally-identical experimental setups, which undercuts treating the taxonomy as a fixed label per task."

Claim: Hu et al. 2026 itself cites this prior taxonomic work as settled background without extending it — its own contribution is a prediction method, not a classification of causes

Hu, Pan, Jhaveri, Lourie & Cho, "Neural Neural Scaling Laws" (arXiv:2601.19831v2, 2026), writes in its Related Work: "individual tasks exhibit diverse behaviors as they scale: some improve monotonically, others plateau, and some even degrade with scale—a phenomenon known as inverse scaling [McKenzie et al., 2023]. Taken together, it seems that no single parametric family can capture the full spectrum of scaling behaviors [Lourie et al., 2025]." This is the full extent of the paper's engagement with which tasks diverge and why: it names the phenomenon, cites the two papers that built causal/behavioral taxonomies of it (McKenzie et al. 2023's four causes; Lourie et al. 2025's six categories), and then pivots to its own contribution — NeuNeu, a neural network that predicts per-task accuracy trajectories without assuming a parametric functional form. Nowhere in the paper does it classify its own 66 evaluation tasks by divergence type or cause; it treats the diversity of scaling behaviors as a prediction target, not an explanatory target.

This directly answers the topic's framing: published ML research goes further than Hu et al. 2026 in characterizing which task types plateau or degrade — but that further work (McKenzie et al. 2023, Wei et al. 2023, Lourie et al. 2025) came before Hu et al. 2026 and is cited by it, rather than being something Hu et al. 2026 itself adds to.

source_url: https://arxiv.org/abs/2601.19831 source_sha: a621d3c50dd274b9c8b909a1a294f3c56f1b137da49b26ece5d865a4753b9bf5 source_title: "Neural Neural Scaling Laws" source_author: "Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie, Kyunghyun Cho" source_date: 2026 source_venue: "New York University; arXiv:2601.19831 (preprint)" source_quote: "individual tasks exhibit diverse behaviors as they scale: some improve monotonically, others plateau, and some even degrade with scale—a phenomenon known as inverse scaling [McKenzie et al., 2023]. Taken together, it seems that no single parametric family can capture the full spectrum of scaling behaviors [Lourie et al., 2025]." source_tier: 1

Seek's commentary: The clean version of the answer is "yes, a taxonomy exists, and it's richer than Hu et al. 2026 lets on" — but it is worth being precise about what kind of taxonomy it is, because it is not quite Heathcote's shape. Heathcote fit a functional form to each individual learner within one task and found the group average distorted the shape everyone actually followed — disaggregation along the subject axis, same task held fixed. McKenzie et al. and Lourie et al. disaggregate along the task axis instead, holding something else (a model family, an experimental setup) comparatively fixed, and classify by the shape or cause of the curve a task's aggregate performance traces across models. Lourie et al.'s own §5 result — the same task, CommonsenseQA, shows nonmonotonic scaling under one experimental setup and a clean line under another — is the detail that keeps this from being a clean Heathcote-style result: Heathcote's individual learners didn't stop being exponential depending on who asked. If the ML taxonomy's categories are this sensitive to setup, it's an open question whether "Distractor Task" or "nonmonotonic" is a property of the task the way "exponential learner" was a property of the subject, or whether it's a property of the task-plus-measurement-choice — closer to an artifact that recurs than a stable natural kind. That question doesn't have a confirmed answer in what was read this session, and the existing vault note on the averaging-artifact analogy (claim-averaging-artifact-across-practice-and-scaling-laws-is-analogous-symptom-not-same-mechanism) already flags the same caution about assuming the two fields' versions of "the aggregate lies" are mechanistically identical. — Seek

Further leads

Caballero, Gupta, Rish & Krueger, "Broken Neural Scaling Laws" (arXiv:2210.14891, ICLR 2023) — proposes a smoothly-broken power-law functional form that can fit non-monotonic curves including double descent and delayed arithmetic jumps; a functional-form fix rather than a causal taxonomy, and used as a baseline (BNSL) inside Hu et al. 2026 itself. Not independently verified this session.
Schaeffer, Schoelkopf, Miranda, Mukobi et al., "Why has predicting downstream capabilities of frontier AI models with scale remained elusive?" (ICML 2025, cited in Lourie et al. 2025) — directly on-topic follow-up question; not fetched this session.
Du, Zeng, Dong & Tang, "Understanding emergent abilities of language models from the loss perspective" (NeurIPS 2024, cited in both Lourie et al. 2025 and Hu et al. 2026) — argues metric choice, not task identity alone, can create or dissolve apparent "emergence"; relevant to whether taxonomy categories are task-intrinsic. Not fetched this session.
Liu, Bertsch, Sutawika et al., "Not-just-scaling laws: Towards a better understanding of the downstream impact of language model design decisions" (EMNLP 2025, cited in Hu et al. 2026) — argues factors beyond compute/loss affect downstream scaling; possibly another taxonomy-adjacent paper from a different lab. Not fetched this session.
Wilcox, Hu, Mueller, Linzen et al., "Bigger is not always better: The importance of human-scale language modeling for psycholinguistics" (2024, cited in Hu et al. 2026 alongside McKenzie et al. 2023 for inverse scaling) — a different field's version of "some tasks get worse with more data/scale"; worth checking whether it adds its own taxonomy or just another data point. Not fetched this session.
A 2026 paper on reliability ceilings (arXiv:2607.14112, found via WebSearch, not fetched/verified) reportedly argues each downstream task has its own "reliability ceiling" that scaling approaches but does not cross for some task types (e.g. creative writing) — possibly a more recent, finer-grained taxonomy candidate; needs primary-source verification before any claim is drawn from it.

Entity candidates

Ian McKenzie — person — lead author of the 2023 Inverse Scaling Prize paper and its four-cause taxonomy; the earlier, foundational work Hu et al. 2026 cites for the very phenomenon it treats as background rather than extending.
Inverse Scaling Prize — concept/event — the 2022 public contest ($250k prize pool, FAR AI / NYU / Anthropic-adjacent organizers) that produced the 11 tasks underlying both McKenzie et al. 2023 and Wei et al. 2023; a reusable dataset (inversescaling.com/data) that recurs across multiple papers in this cluster.
Nicholas Lourie — person — co-author of the 2025 six-category taxonomy paper and co-author (with Hu) of Hu et al. 2026 itself; the direct predecessor whose own prior work already answers most of this topic's question.
Jason Wei — person — found that inverse-scaling tasks mostly reverse into U-shaped scaling at larger model sizes, and proposed the distractor-task mechanism for why; also lead author of the separate, widely-cited "Emergent Abilities of Large Language Models" (TMLR 2022) that this cluster's papers repeatedly cite as the companion phenomenon to inverse scaling.
Michael Y. Hu — person — co-author of both Lourie et al. 2025 and Hu et al. 2026; the continuity between the two papers (same lead author, one year apart) is itself part of the answer to this topic.
Kyunghyun Cho — person — senior co-author on both Lourie et al. 2025 and Hu et al. 2026; already present in the vault via entity-jared-kaplan-adjacent scaling-law material but not yet given his own entity page.
distractor task — concept/term — the proposed mechanism (a model learns an easier wrong sub-task before learning the intended harder one) offered by both McKenzie et al. 2023 and Wei et al. 2023 to explain scale-dependent reversal; recurring enough across this cluster to warrant its own note.
U-shaped scaling — concept/term — the specific divergence pattern (performance drops then recovers with scale) that complicates treating "plateau/degrade" as a fixed per-task label.

written by claude-sonnet-5 · Batch research capture, 2026-10-23 · raw markdown