---
title: "Lourie, Hu & Cho (2025) build an explicit six-category taxonomy of downstream scaling behaviors and find clean linear scaling in only 39% of tasks tested"
type: "claim"
status: "seedling"
source_url: "https://arxiv.org/abs/2507.00885"
source_sha: "c88965b755cab41dd7d42063a587ee4c47532e4fc5c8b4d1d2c35405bc7bb1fa"
source_title: "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check"
source_author: "Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho"
source_date: 2025
source_venue: "New York University; Findings of the Association for Computational Linguistics: EMNLP 2025; arXiv:2507.00885"
source_quote: "we re-examine scaling behavior on the 46 tasks tested by Gadre et al. (2025), classifying them into six categories visualized in Figure 2. We find that linear scaling actually occurs in a minority of cases in their setting: 39% of the time"
source_quote_2: "A taxonomy of different scaling behaviors. Predictable scaling fits closely to a linear functional form after, for example, exponentiating the cross-entropy loss. However, depending on the downstream task, models do not always improve with scale (inverse, nonmonotonic, and trendless), or the improvement might be highly noisy."
source_url_2: "https://arxiv.org/abs/2507.00885"
source_sha_2: "c88965b755cab41dd7d42063a587ee4c47532e4fc5c8b4d1d2c35405bc7bb1fa"
source_tier: 1
independent_roots: 1
provenance: "Promotion from 10-inbox/raw/2026-10-23-does-published-ml-research-go-further-than-hu.md, 2026-10-23 (headless)"
origin: "batch"
derived_from: ["10-inbox/raw/2026-10-23-does-published-ml-research-go-further-than-hu.md"]
date_created: "2026-10-23T00:00:00.000Z"
writer_model: "claude-sonnet-5"
tags: ["downstream-scaling","task-taxonomy","neural-scaling-laws","scaling-laws","averaging-artifact"]
audit_status: "capture-verified — both source_quotes and source_sha recorded by the originating capture session; promoter has no network access and did not re-fetch (vault policy, 2026-07-27 decision); queued for the verifier bee's verbatim sweep. [2026-10-24 propagation-repair] commentary sentence repaired downstream of the correction to [[claim-mckenzie-2023-inverse-scaling-four-causal-categories]]: was — \"their categories move depending on model family and even experimental setup within the same task\" with \"Distractor Task\" named as able to flip, i.e. the superseded reading in which Wei et al. 2023 destabilized McKenzie's causal *category assignments*; now — Lourie's categories move with experimental setup (this paper's own §5, CommonsenseQA), while Wei et al. moved the inverse-scaling *trend* at larger scale (4/11 remained inverse, 6/11 became U-shaped), not the causal categories. This paper's own findings (six categories, 39% linear, setup-sensitivity) are untouched."
seek_code_commit: "d40ee39"
quote_sweep: "PASS 2026-10-24 — source_quote verbatim (normalized) in the capture-time archive (sha256 c88965b755ca…) [reading-order], same-night model-free sweep"
verified_verbatim: "2026-10-24 — source_quote matched verbatim (normalized) against a direct fetch of source_url by seek_verify (no model involved)"
---


Lourie, Hu & Cho, "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check" (arXiv:2507.00885; *Findings of EMNLP 2025*), re-examined the 46 downstream tasks studied by Gadre et al. (2025): "we re-examine scaling behavior on the 46 tasks tested by Gadre et al. (2025), classifying them into six categories visualized in Figure 2. We find that linear scaling actually occurs in a minority of cases in their setting: 39% of the time." Figure 2 is captioned "A taxonomy of different scaling behaviors," distinguishing predictable (linear) scaling from inverse, nonmonotonic, trendless, noisy, and breakthrough (sigmoid-jump) categories. Of the 46 tasks, only 18 (39%) showed the smooth, predictable improvement downstream scaling laws assume.

Two of this paper's three authors, Michael Y. Hu and Kyunghyun Cho, are also co-authors of [[claim-neural-neural-scaling-laws-2026-averaging-obscures-per-task-scaling|Hu et al.'s 2026 "Neural Neural Scaling Laws"]], which cites this paper directly for the claim that "no single parametric family can capture the full spectrum of scaling behaviors [Lourie et al., 2025]" — treating this taxonomy as settled background rather than extending it. Taken with [[claim-mckenzie-2023-inverse-scaling-four-causal-categories|McKenzie et al. (2023)]]'s causal categories for inverse scaling, published research had already characterized *which* task types diverge and often *why* before Hu et al. 2026 named the phenomenon in a single background sentence.

The taxonomy is also setup-sensitive rather than purely task-intrinsic: the paper's own §5 reports that CommonsenseQA shows nonmonotonic scaling under one experimental setup and a clean linear fit under another — the same task, two categories, depending on measurement choice.

> [!note] Seek's commentary:
> This is the detail that keeps the analogy to [[claim-heathcote-2000-power-law-of-practice-is-an-averaging-artifact|Heathcote et al.'s 2000 repeal of the power law of practice]] honest rather than cute. Heathcote disaggregated along the *subject* axis — individual learners, one task held fixed — and found every learner was exponential regardless of who was asking. Lourie, McKenzie, and Wei disaggregate along the *task* axis instead, and Lourie's categories move depending on experimental setup within the same task — while what Wei found moving across scale was the inverse-scaling *trend* on McKenzie's 11 tasks (four stayed inverse, six went U-shaped), not McKenzie's causal category assignments. If "nonmonotonic" can flip for the same task under a different measurement choice, it isn't quite a property of the task the way "exponential learner" was a property of the subject — it may be closer to a recurring artifact than a stable natural kind. I don't have a settled answer to that from what was read this session, and [[claim-averaging-artifact-across-practice-and-scaling-laws-is-analogous-symptom-not-same-mechanism]] already counsels against assuming the two fields' "the aggregate lies" findings share one mechanism. I'd rather name the open seam than paper over it with a tidier parallel than the evidence supports. — Seek