all tags

neural-scaling-laws

15 notes carry this tag — 6 claims, 2 observations, 2 questions, 4 captures, 1 essay. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

Essays

• 
  The line no one walks
  essay draft 2026-07-18 · learning-curve, scaling-laws, wrights-law, averaging-artifact, history-of-psychology

Everything else

• What are the actual neural-scaling-law exponents in Kaplan et al. (2020) and Hoffmann/Chinchilla (2022), and do they support a 'power-law diminishing returns' reading?
  capture promoted Tier 1 2026-08-22 · neural-scaling-laws, scaling-laws, chinchilla, hoffmann-2022, kaplan-2020
• Chinchilla (70B parameters, 1.4T tokens, same compute as Gopher) uniformly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)
  claim seedling Tier 1 2026-08-22 · neural-scaling-laws, chinchilla, hoffmann-2022, compute-optimal-training, benchmark
• Hoffmann et al. (2022) find compute-optimal LLM training splits equally between model size and training tokens (N_opt, D_opt ∝ C^~0.5), contradicting Kaplan et al.'s (2020) 0.73/0.27 split
  claim seedling Tier 1 2026-08-22 · neural-scaling-laws, chinchilla, hoffmann-2022, compute-optimal-training, kaplan-2020
• Hoffmann et al.'s (2022) own loss-decay exponents (α=0.34 on parameters, β=0.28 on data) are roughly 3–4.5x larger than the exponents Kaplan et al. (2020) reported for the same relationship
  claim seedling Tier 1 2026-08-22 · neural-scaling-laws, chinchilla, hoffmann-2022, kaplan-2020, diminishing-returns
• Heathcote (2000) and Hu et al. (2026) share a symptom — averaging conceals heterogeneity — but not a statistical mechanism
  claim seedling Tier 1 2026-08-01 · cross-domain-bridge, cross-time-bridge, averaging-artifact, power-law, scaling-laws
• Kaplan et al. (2020) neural-scaling-law exponents are small (~0.05–0.095), meaning 'arbitrary' scaling is real but logarithmically diminishing
  claim seedling Tier 1 2026-07-28 · neural-scaling-laws, scaling-laws, kaplan-2020, diminishing-returns, quantitative
• The Bitter Lesson names the hand-designed-to-learned pattern the myth ledger documents, but Sutton's 'scale arbitrarily' misses the logarithmic brake note 2 supplies
  capture promoted Tier 1 2026-07-27 · cross-domain-bridge, bitter-lesson, scaling-laws, neural-scaling-laws, convolutional-networks
• Do Heathcote et al. (2000) and 'Neural Neural Scaling Laws' (2026) describe the literal same averaging mechanism, or only an analogous symptom?
  capture promoted 2026-07-26 · power-law, averaging-artifact, scaling-laws, cognitive-psychology, neural-scaling-laws
• Aggregate validation loss follows a smooth power law while individual downstream tasks scale diversely — averaging obscures the per-task signal (Neural Neural Scaling Laws, 2026)
  claim seedling Tier 1 2026-07-25 · neural-scaling-laws, scaling-laws, averaging-artifact, deep-learning, downstream-tasks
• Averaging over heterogeneous learners can manufacture a smooth power law that no individual component obeys — a 2000 cognitive-psychology critique and a 2026 neural-scaling-law paper name the same artifact
  observation seedling Tier 1 2026-07-25 · cross-domain-bridge, cross-time-bridge, averaging-artifact, power-law, scaling-laws
• Do Heathcote et al. (2000) and 'Neural Neural Scaling Laws' (2026) describe the literal same averaging mechanism, or only an analogous symptom?
  question answered 2026-07-25 · averaging-artifact, power-law, scaling-laws, cognitive-psychology, neural-scaling-laws
• The power law of practice was repealed as an averaging artifact of exponential individual learners — and 2026 neural-scaling-law papers report the identical artifact
  capture promoted Tier 1 2026-07-16 · power-law, learning-curve, scaling-laws, cognitive-psychology, neural-scaling-laws
• Population-scaled adaptive improvement hits the same sub-linear (logarithmic / power-law) brake across evolution, idea-production, and neural scaling
  observation seedling Tier 1 2026-07-12 · cross-domain-bridge, scaling-laws, diminishing-returns, endogenous-growth, population-genetics
• What are the actual neural-scaling-law exponents in Kaplan et al. (2020) and Hoffmann/Chinchilla (2022), and do they support a 'power-law diminishing returns' reading?
  question answered 2026-07-12 · neural-scaling-laws, verification, ai, diminishing-returns, kaplan-2020