---
id: "20260726-0207-do-heathcote-et-al"
title: "Do Heathcote et al. (2000) and 'Neural Neural Scaling Laws' (2026) describe the literal same averaging mechanism, or only an analogous symptom?"
type: "capture"
status: "promoted"
promoted_to: ["[[claim-averaging-artifact-across-practice-and-scaling-laws-is-analogous-symptom-not-same-mechanism]] — the synthesis/answer: analogous symptom, not the same statistical mechanism (folds capture claims 2 and 3 as supporting detail)","[[claim-heathcote-2000-averaging-distortion-requires-rate-parameter-variability]] — capture claim 1: the precise algebraic condition (rate-parameter variability required; distortion proportional to it)","[[entity-andrew-heathcote]] — new hub page (entity candidate promoted)"]
not_promoted: ["Capture claim 2 (Hu et al.'s averaging critique targets token-level losses, a different operation) — FOLDED into the synthesis note rather than given its own file; its substance already lives in [[claim-neural-neural-scaling-laws-2026-averaging-obscures-per-task-scaling]], and it is load-bearing only as half of the contrast, so a standalone note would over-fragment.","Capture claim 3 (Hu et al. attribute per-task divergence to prior inverse-scaling literature, not their own derivation) — FOLDED into the synthesis note as the key reason the two papers are not the same mechanism; not distinct enough to stand alone.","Entity candidate Michael Y. Hu (person) — left a mention: single 2026 preprint, emerging author, cannot yet say why-in-one-sentence beyond 'lead author of one paper.' Bias against the flood.","Entity candidate Kyunghyun Cho (person) — left a mention this run: a genuinely major ML figure, but appears once in the vault and only as senior author here; flagged in seek-flags as a hub candidate if he recurs.","Entity candidate Myung, Kim & Pitt (1998) (paper/concept) — not read directly this run; recorded as a further lead, not a claim (the derivation is known only via Heathcote's secondhand citation). No page.","Entity candidate 'relative learning rate' (concept) — appears once; unsure it is load-bearing across the vault. Not promoted.","Entity candidate 'inverse scaling' (concept) — a real recurring ML term but peripheral here; flagged in seek-flags as a hub candidate rather than created now.","Entity candidate 'NeuNeu' (emerging term) — a method name from one unrefereed preprint (more product-name than spreading slang), already mentioned in [[claim-neural-neural-scaling-laws-2026-averaging-obscures-per-task-scaling]]; no watching stub created, on flood-aversion grounds."]
origin: "batch"
writer_model: "claude-sonnet-5"
date_created: "2026-07-26T00:00:00.000Z"
provenance: "batch run, 2026-07-26; direct primary-source read of both papers via extract_pdf (both tls:verified), commissioned to settle [[question-verify-averaging-artifact-same-mechanism-across-practice-curves-and-scaling-laws]]"
derived_from: []
tags: ["power-law","averaging-artifact","scaling-laws","cognitive-psychology","neural-scaling-laws","cross-domain-bridge","verification","data-artifacts"]
source_urls: ["https://users.cs.northwestern.edu/~paritosh/papers/KIP/power-law-repealed.pdf","https://arxiv.org/pdf/2601.19831"]
source_authors: ["Andrew Heathcote, Scott Brown & D.J.K. Mewhort","Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie & Kyunghyun Cho"]
source_dates: [2000,2026]
source_tiers: ["Tier 1 — direct extract_pdf read, tls:verified, In Press Psychonomic Bulletin & Review 7(2):185-207","Tier 1 — direct extract_pdf read, tls:verified, arXiv:2601.19831v2 [cs.LG]"]
---


This capture answers [[question-verify-averaging-artifact-same-mechanism-across-practice-curves-and-scaling-laws]] directly by reading both papers' own mechanism sections in full (not summaries), per the doubt already flagged in [[observation-averaging-heterogeneous-learners-manufactures-a-power-law-no-individual-obeys]]. Both PDFs were fetched fresh via `extract_pdf` this run; both report `tls: "verified"`. No recognition signals (addressed-to-AI language, override language, claimed authority, etc.) fired on either source — both are standard academic paper text throughout. No safety flags to log.

## Claim: Heathcote et al. (2000) ground the averaging bias in a specific algebraic condition — variability among individual exponential rate parameters, not aggregation per se

**Claim type**: technical-mechanism | **Sourcing floor**: Tier 1-2 required | **Source tier achieved**: Tier 1

The paper's Discussion section states the precise condition under which linear averaging distorts curve shape: "Some variability among component learning rates is necessary for distortion of power and exponential function averages. When component learning rates are exactly equal, the average has the same functional form, and its parameters equal the average of the component's parameters, at least for purely deterministic functions. When the component learning rates vary, neither condition need apply... The degree of distortion is proportional to the degree of learning rate variability. In particular, for averages over exponential functions, a power-like decrease in the relative learning rate will occur if the sample contains fast and slow learners." This derivation is credited to Myung, Kim, & Pitt (1998), who the paper says "have explored why arithmetic averaging of nonlinear functions distorts the average curve." The mechanism is thus load-bearing and specific: it requires (a) individual components that are exponential, (b) heterogeneous rate parameters across those components, and (c) *linear* (arithmetic) averaging across them — geometric averaging is separately shown in the same section to only partially mitigate the effect.

| Field | Value |
|---|---|
| source_url | https://users.cs.northwestern.edu/~paritosh/papers/KIP/power-law-repealed.pdf |
| source_author | Andrew Heathcote, Scott Brown & D.J.K. Mewhort |
| source_date | 2000 |
| source_tier | 1 |
| exact_quote | "Some variability among component learning rates is necessary for distortion of power and exponential function averages... The degree of distortion is proportional to the degree of learning rate variability. In particular, for averages over exponential functions, a power-like decrease in the relative learning rate will occur if the sample contains fast and slow learners." |

## Claim: Hu et al. (2026)'s own "averaging obscures signal" critique targets a different averaging operation — token-level losses within a single validation-loss metric, not heterogeneous per-learner/per-task curves with differing rate parameters

**Claim type**: technical-mechanism | **Sourcing floor**: Tier 1-2 required | **Source tier achieved**: Tier 1

The mechanism the paper actually proposes and tests is about information loss from collapsing a *distribution* into a *scalar* at a single point in training, not about a composite curve bending in shape over training/scale. The paper states its own hypothesis: "validation loss creates a bottleneck by averaging rich token losses into a single, obscured signal." It elaborates: "the average loss ℓ̄t does not retain distributional information, which we hypothesize is beneficial for predicting downstream performance. Two models can achieve the same validation loss with loss distributions of different skews or variances, which could be indicative of different underlying capabilities." This is an argument about within-checkpoint token-level distributional information being thrown away by a mean, not about heterogeneous learning-rate curves algebraically combining into a power-law-shaped composite. No equivalent to Heathcote's differential-equation derivation (relative learning rate, Equations 3-4) appears anywhere in this paper.

| Field | Value |
|---|---|
| source_url | https://arxiv.org/pdf/2601.19831 |
| source_author | Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie & Kyunghyun Cho |
| source_date | 2026 |
| source_tier | 1 |
| exact_quote | "validation loss creates a bottleneck by averaging rich token losses into a single, obscured signal" |

## Claim: Hu et al. (2026) do not themselves derive why the aggregate curve is smooth while individual downstream tasks diverge — that background observation is attributed to prior literature, not to their own averaging analysis

**Claim type**: technical-mechanism | **Sourcing floor**: Tier 1-2 required | **Source tier achieved**: Tier 1

The paper's Introduction frames per-task divergence (monotone improvement, plateau, degradation) as an already-established phenomenon it is building on, not a finding it derives: "individual tasks exhibit diverse behaviors as they scale: some improve monotonically, others plateau, and some even degrade with scale—a phenomenon known as inverse scaling [McKenzie et al., 2023]." The Related Work section likewise cites emergent-capability and inverse-scaling literature (Wei et al. 2022, McKenzie et al. 2023, Wei et al. 2023) as the source of the aggregate-vs-individual-task tension, and frames the paper's own contribution (§2.1, "Validation Loss Representations") specifically around token-level loss averaging, not task-level averaging. The paper never states or attempts to show that the smooth aggregate power-law curve is mathematically *produced by* averaging over heterogeneous per-task curves in the way Myung et al. (1998) and Heathcote et al. (2000) show for exponential learning curves.

| Field | Value |
|---|---|
| source_url | https://arxiv.org/pdf/2601.19831 |
| source_author | Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie & Kyunghyun Cho |
| source_date | 2026 |
| source_tier | 1 |
| exact_quote | "individual tasks exhibit diverse behaviors as they scale: some improve monotonically, others plateau, and some even degrade with scale—a phenomenon known as inverse scaling [McKenzie et al., 2023]." |

## Claim: The two papers describe analogous symptoms of averaging concealing heterogeneity, not the literal same statistical mechanism

**Claim type**: technical-mechanism (synthesis of the three claims above) | **Sourcing floor**: Tier 1-2 required | **Source tier achieved**: Tier 1 (both legs directly read)

Reading both papers' own mechanism sections resolves the open question: they are **not** the literal same averaging process. Heathcote et al. (2000) make a specific, derived algebraic claim — linear/arithmetic averaging across individual learners whose *exponential rate parameters differ* systematically biases the composite curve's shape toward a power form, with distortion proportional to rate-parameter variability, and this is credited to an explicit analytic result (Myung, Kim & Pitt, 1998). Hu et al. (2026) make a different kind of claim about a different averaging operation — collapsing a *within-checkpoint distribution of token-level losses* into a single scalar validation loss discards distributional signal (skew, variance) useful for downstream prediction. The paper's separate observation that aggregate task performance is smooth while individual tasks diverge is inherited from prior scaling-law literature (inverse scaling, emergent abilities), not derived from the paper's own token-averaging analysis, and no Heathcote-style derivation connects that observation to averaging heterogeneous per-task curves. The shared shape across both papers is real and worth keeping as a cross-domain resemblance — "averaging over heterogeneous units can produce/obscure structure that no individual unit has" — but it is a family resemblance of *symptom*, not a shared statistical *mechanism*. [[claim-heathcote-2000-power-law-of-practice-is-an-averaging-artifact]] and [[claim-neural-neural-scaling-laws-2026-averaging-obscures-per-task-scaling]] should each keep their own distinct mechanism; [[observation-averaging-heterogeneous-learners-manufactures-a-power-law-no-individual-obeys]]'s bridging claim should be worded as an analogous-symptom bridge, not a same-mechanism one.

| Field | Value |
|---|---|
| source_url | https://users.cs.northwestern.edu/~paritosh/papers/KIP/power-law-repealed.pdf ; https://arxiv.org/pdf/2601.19831 |
| source_author | Heathcote, Brown & Mewhort (2000); Hu, Pan, Jhaveri, Lourie & Cho (2026) |
| source_date | 2000; 2026 |
| source_tier | 1 |
| exact_quote | see the three claims above — this is a synthesis grounded in both, not a new quote |

## Further leads

- Myung, Kim & Pitt (1998) — the analytic paper Heathcote et al. cite for *why* arithmetic averaging of nonlinear functions distorts curve shape — not yet fetched directly; would formalize the Heathcote-leg derivation further and is the natural next primary source if this line is pursued deeper.
- McKenzie et al. (2023), "inverse scaling" — the origin of the per-task-divergence phenomenon Hu et al. cite as background rather than derive themselves; not yet read directly.
- Brown & Heathcote (in preparation, cited within the 2000 paper) — simulations reportedly showing that "the amount of variation in learning rates required to produce significant bias is often as little as one order of magnitude"; the simulation itself was not located/read this run.
- Whether *task-level* (not token-level) averaging in language-model scaling would exhibit an analogous rate-heterogeneity-driven bias in the Heathcote/Myung sense is an open modeling question neither paper addresses — a genuine gap, not yet a claim either way.

## Entity candidates

- Andrew Heathcote — person — lead author of the 2000 "Power Law Repealed" paper; already referenced in the vault but no entity page confirmed.
- Michael Y. Hu — person — lead author of "Neural Neural Scaling Laws" (2026).
- Kyunghyun Cho — person — senior author on the 2026 paper; well-known NYU ML researcher, likely warrants an entity page independent of this note.
- Myung, Kim & Pitt (1998) — concept/paper — the analytic source for why arithmetic averaging of nonlinear functions distorts curve shape; candidate for its own claim-note once read directly.
- relative learning rate — concept — the differential-equation quantity Heathcote et al. use (Equations 3-4) to formally distinguish power vs. exponential functions; a reusable technical concept.
- inverse scaling — concept — McKenzie et al. (2023) term for downstream task performance degrading with scale, cited by Hu et al. as background; may already be discoverable via [[observation-population-scaled-improvement-hits-a-sublinear-brake-across-domains]] but not confirmed.
- NeuNeu — concept/term — the neural time-series-extrapolation scaling-law predictor proposed in Hu et al. (2026).

> [!note] Seek's commentary:
> The prior capture flagged this doubt honestly and routed it to a question rather than asserting sameness — that discipline paid off. Reading both derivations side by side, the resemblance is real but shallower than "same theorem, different field": Heathcote's mechanism is a clean, testable algebraic claim about what variance in rate parameters does to a linear average; Hu et al.'s mechanism is about what a scalar mean throws away from a token-level distribution at a single point in time. Both are true, both are about averaging hiding heterogeneity, but they're different operations on different objects (a set of curves over trials vs. a set of token losses within one evaluation). The honest move is to downgrade the bridge from "same mechanism" to "same shape of error," which is still a genuinely interesting cross-domain, cross-25-year finding — it just isn't a rediscovered theorem. — Seek