---
title: "The vanishing gradient problem is a chain-rule pathology — early layers stop learning as gradient magnitudes diverge across depth"
type: "claim"
status: "budding"
audit_status: "capture-verified (definitional claim on Wikipedia T3 within floor; Bengio-1994 bibliographic anchor via PubMed T2 at capture level)"
source_url: "https://en.wikipedia.org/wiki/Vanishing_gradient_problem"
source_title: "Vanishing gradient problem (Wikipedia)"
source_author: "Wikipedia contributors (definition); Bengio, Simard & Frasconi 1994, IEEE Trans. Neural Networks 5(2):157-166 (canonical characterization, PMID 18267787)"
source_date: "accessed 2026-07-01"
source_tier: 3
provenance: "Promotion from 10-inbox/raw/2026-07-01-what-is-the-vanishing-gradient-problem-...md, 2026-07-06, queen cycle 7"
origin: "session"
date_created: "2026-07-06T00:00:00.000Z"
tags: ["vanishing-gradient","backpropagation","chain-rule","deep-learning","training-pathology"]
drafted_in: ["2026-07-13-magnet-under-the-transformer","magnet-under-the-transformer"]
---


The vanishing gradient problem is "the problem of greatly diverging gradient
magnitudes between earlier and later layers encountered when training neural
networks with backpropagation": as the loss signal traverses many layers
backward, early-layer weight updates shrink toward zero while later layers
keep updating — the layers that learn basic representations effectively stop
learning. The inverse pathology (exploding gradients) arises from the same
structure: repeated chain-rule multiplication of Jacobians across depth.

This is the failure mode *inside* the algorithm the origins cluster traces:
[[entity-backpropagation|the backward pass]] ([[claim-backpropagation-special-case-of-reverse-mode-ad]])
multiplies per-layer derivatives, so whatever is slightly less than or
greater than one compounds exponentially with depth. It is a large part of
why deep networks stalled for a decade after 1986 and why the field's
fixes (ReLU, normalization, residual connections, LSTM gating) all target
the multiplication chain.

Attribution boundary, honestly held: the field's canonical peer-reviewed
characterization is Bengio, Simard & Frasconi (1994, bibliographic record
Tier-2-confirmed); the claim that Hochreiter's 1991 Diplom thesis was the
*first* formal identification circulates widely but the capture found no
independent Tier 1–2 source asserting primacy — that sub-claim stays in the
capture under its [unverified-historical] flag. See
[[moc-backpropagation-origins]], [[claim-rhw-1986-demonstration-not-invention]].