---
title: "Capture: What is the reverse-mode automatic differentiation / backpropagation correspondence, stated precisely?"
type: "capture"
status: "promoted"
date_promoted: "2026-07-07T00:00:00.000Z"
promoted_to: ["30-notes/claim-backpropagation-special-case-of-reverse-mode-ad.md"]
not_promoted: ["Two-phase forward/backward sweep mechanism — already covered by claim-linnainmaa-reverse-mode-single-pass (retrieve-before-write collision; linked instead)","Many-inputs/few-outputs efficiency condition — folded into the promoted note and the Linnainmaa cluster; the cheap-gradient numeric bound stays [unverified-quant] pending Griewank & Walther 2008 (queued topic)","Reverse-mode-more-general-than-backprop claim — folded into the promoted note's direction-matters paragraph"]
origin: "batch"
date_created: "2026-06-29T00:00:00.000Z"
provenance: "batch run 2026-06-29 — researched via web search/fetch; primary source for the load-bearing mechanism and correspondence claims is Baydin, Pearlmutter, Radul & Siskind, 'Automatic Differentiation in Machine Learning: a Survey' (JMLR 2018 / arXiv:1502.05767), a peer-reviewed survey paper"
tags: ["automatic-differentiation","reverse-mode-ad","backpropagation","chain-rule","computational-graph","adjoint","gradient-descent"]
source_url: "https://jmlr.org/papers/v18/17-468.html"
source_author: "Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind"
source_date: "2018 (JMLR 18(153):1–43); arXiv preprint 1502.05767 first posted 2015"
source_tier: 1
other_sources: [{"url":"https://en.wikipedia.org/wiki/Automatic_differentiation","note":"definitional/corroborating source, accessed 2026-06-29","tier":3}]
---


This capture answers a precise version of a claim already gestured at in [[backpropagation-gap]] and the Linnainmaa cluster of notes (e.g. [[claim-linnainmaa-reverse-mode-single-pass]], [[claim-linnainmaa-priority-not-paternity]]): that backpropagation "is" reverse-mode automatic differentiation. The vault has so far recorded this as a historical/identity claim resting partly on Wikipedia (Tier 3) and flagged `[unverified-mechanism]` pending a primary source. This capture grounds the mechanism claim itself in a Tier 1 peer-reviewed source — Baydin et al.'s JMLR survey, which exists specifically to "precisely defin[e] the main differentiation techniques and their interrelationships" (its own stated purpose, per the abstract) — so the correspondence can be stated exactly rather than approximately.

---

## Claim: Backpropagation is the special case of reverse-mode automatic differentiation applied to a neural network's forward computation composed with a scalar loss function

**Claim type:** technical-mechanism — Tier 1–2 required. Achieved Tier 1.

The standard description of backpropagation — propagate the loss's sensitivity backward from the output, using the chain rule to get the gradient with respect to each weight — is, precisely, what results from applying reverse-mode automatic differentiation to one specific composite function: the network's evaluation function composed with its scalar objective (loss) function. Baydin et al. state this directly:

> "In simplest terms, backpropagation models learning as gradient descent in neural network weight space, looking for the minima of an objective function. The required gradient is obtained by the backward propagation of the sensitivity of the objective value at the output... utilizing the chain rule to compute partial derivatives of the objective with respect to each weight. The resulting algorithm is essentially equivalent to transforming the network evaluation function composed with the objective function under reverse mode AD, which, as we shall see, actually generalizes the backpropagation idea."

The same paper restates the correspondence directly at the definition of reverse mode itself:

> "AD in the reverse accumulation mode corresponds to a generalized backpropagation algorithm, in that it propagates derivatives backward from a given output."

This is corroborated by Wikipedia's "Automatic differentiation" article (Tier 3, definitional claim — within floor): "Backpropagation of errors in multilayer perceptrons, a technique used in machine learning, is a special case of reverse accumulation." The direction of the relationship is one-way: backpropagation is an instance of reverse-mode AD (the narrower, neural-network-specific name), not a synonym for it — see the next claim.

**Provenance:**
- source_url: https://jmlr.org/papers/v18/17-468.html (PDF: https://jmlr.org/papers/volume18/17-468/17-468.pdf; arXiv mirror: https://arxiv.org/abs/1502.05767)
- source_author: Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind
- source_date: 2018, JMLR 18(153):1–43
- source_tier: 1
- exact quote: "The resulting algorithm is essentially equivalent to transforming the network evaluation function composed with the objective function under reverse mode AD, which, as we shall see, actually generalizes the backpropagation idea." / "AD in the reverse accumulation mode corresponds to a generalized backpropagation algorithm, in that it propagates derivatives backward from a given output."
- corroborating source: Wikipedia, "Automatic differentiation," https://en.wikipedia.org/wiki/Automatic_differentiation (accessed 2026-06-29). Tier 3. Quote: "Backpropagation of errors in multilayer perceptrons, a technique used in machine learning, is a special case of reverse accumulation."
- related existing notes: [[backpropagation-gap]], [[claim-linnainmaa-priority-not-paternity]]

---

## Claim: Reverse-mode AD computes derivatives in two phases — a forward sweep that evaluates and records intermediate variables, then a backward sweep that propagates "adjoints" (the partial derivative of the single output with respect to each intermediate variable) via the chain rule from output back to inputs

**Claim type:** technical-mechanism — Tier 1–2 required. Achieved Tier 1.

The object reverse-mode AD actually computes and passes backward is the *adjoint*: for a chosen scalar output y and any intermediate variable vᵢ in the computation, the adjoint is v̄ᵢ = ∂y/∂vᵢ — "the sensitivity of a considered output yⱼ with respect to changes in vᵢ" (Baydin et al.). The algorithm runs in two distinct phases:

> "In the first phase, the original function code is run forward, populating intermediate variables... In the second phase, derivatives are calculated by propagating adjoints in reverse, from the outputs to the inputs."

Wikipedia's definitional account (Tier 3, consistent with the Tier 1 description above) gives the same structure with explicit notation: the adjoint is "a derivative of a chosen dependent variable with respect to a subexpression wᵢ: w̄ᵢ = ∂y/∂wᵢ," computed via the recursive relation ∂y/∂wᵢ = (∂y/∂wᵢ₊₁)(∂wᵢ₊₁/∂wᵢ), traversing "from outside to inside" — i.e., from the output back toward the inputs. This is the general mechanism of which backpropagation's "forward pass, then backward pass of error signals" is the neural-network instance: in backprop, the single scalar output being differentiated is the loss/error term, so "y would be a scalar corresponding to the error E" (Baydin et al.).

This generalizes the single-backward-pass structure already recorded for Linnainmaa's 1970 algorithm in [[claim-linnainmaa-reverse-mode-single-pass]] (many local rounding errors, one accumulated-error output, all sensitivities recovered in one backward sweep) — the same mathematical shape as many weights, one loss.

**Provenance:**
- source_url: https://jmlr.org/papers/v18/17-468.html (arXiv: https://arxiv.org/abs/1502.05767, Section 3.2 "Reverse Mode")
- source_author: Baydin, Pearlmutter, Radul, Siskind
- source_date: 2018
- source_tier: 1
- exact quote: "complementing each intermediate variable vᵢ with an adjoint v̄ᵢ = ∂yⱼ/∂vᵢ, which represents the sensitivity of a considered output yⱼ with respect to changes in vᵢ"; "In the first phase, the original function code is run forward, populating intermediate variables... In the second phase, derivatives are calculated by propagating adjoints in reverse, from the outputs to the inputs."; "In the case of backpropagation, y would be a scalar corresponding to the error E."
- corroborating source: Wikipedia, "Automatic differentiation," https://en.wikipedia.org/wiki/Automatic_differentiation (accessed 2026-06-29). Tier 3. Quotes: "Reverse accumulation traverses from outside to inside"; adjoint defined as "w̄ᵢ = ∂y/∂wᵢ."
- related existing notes: [[claim-linnainmaa-reverse-mode-single-pass]], [[claim-linnainmaa-rounding-error-problem]]

---

## Claim: Reverse-mode AD is the efficient choice specifically when a function has many inputs and few (ideally one) outputs — exactly the shape of neural-network training, where there are many weights and one scalar loss — which is why it requires roughly one backward sweep regardless of the number of inputs

**Claim type:** technical-mechanism — Tier 1–2 required. Achieved Tier 1.

Baydin et al. note the asymmetry directly when introducing forward mode (the alternative AD strategy): for a function f: ℝⁿ → ℝᵐ, "In general, for cases f: ℝⁿ → ℝᵐ where n ≫ m, a different technique is often preferred" — that technique being reverse mode, developed in the following section. Wikipedia's parallel, more explicit statement (Tier 3, consistent with the Tier 1 source) quantifies the asymmetry directly: "Reverse accumulation is more efficient than forward accumulation for functions f : ℝⁿ → ℝᵐ with n ≫ m as only m sweeps are necessary, compared to n sweeps for forward accumulation."

A neural network's training loss is exactly this shape: n is the number of weights (potentially billions), m is 1 (a single scalar loss). Reverse mode therefore recovers the gradient with respect to every weight in roughly one backward sweep — a cost described qualitatively by Baydin et al. as adding only "a small constant factor of overhead" over the forward evaluation itself, rather than scaling with the number of weights. (No specific multiplier, e.g. "2x" or "3x," appears in this source; a precise constant-factor figure is not recorded here and would need a separate primary source, e.g. Griewank & Walther's *Evaluating Derivatives*, which formulates the related "cheap gradient principle" — see Further leads.)

This many-inputs/one-output structure is the same structure already established for Linnainmaa's 1970 rounding-error problem in [[claim-linnainmaa-rounding-error-problem]] ("many inputs (N local rounding errors), one output (the accumulated error)... exactly the regime in which a backward propagation of derivatives is efficient").

**Provenance:**
- source_url: https://jmlr.org/papers/v18/17-468.html (arXiv: https://arxiv.org/abs/1502.05767, Section 3.1 "Forward Mode" / Section 3.2 "Reverse Mode")
- source_author: Baydin, Pearlmutter, Radul, Siskind
- source_date: 2018
- source_tier: 1
- exact quote: "In general, for cases f:ℝⁿ→ℝᵐ where n≫m, a different technique is often preferred."; "This allows accurate evaluation of derivatives at machine precision with only a small constant factor of overhead and ideal asymptotic efficiency."
- corroborating source: Wikipedia, "Automatic differentiation," https://en.wikipedia.org/wiki/Automatic_differentiation (accessed 2026-06-29). Tier 3. Quote: "Reverse accumulation is more efficient than forward accumulation for functions f : R^n → R^m with n ≫ m as only m sweeps are necessary, compared to n sweeps for forward accumulation."
- related existing notes: [[claim-linnainmaa-rounding-error-problem]]

---

## Claim: Reverse-mode AD is strictly more general than backpropagation — it differentiates arbitrary programs (branching, loops, recursion) via the chain rule over elementary operations, not only the layered structure of a feedforward network — which is why "backpropagation" names the special case and not the general method

**Claim type:** definitional — Tier 3–4 acceptable; achieved Tier 1.

The relationship is asymmetric, and the asymmetry is the precise content of the correspondence. Baydin et al. open their survey by stating it as the paper's organizing distinction:

> "Automatic differentiation (AD), also called algorithmic differentiation or simply 'autodiff', is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs."

AD (and reverse mode specifically) applies to any composition of elementary differentiable operations expressible as code — "allowing branching, loops, and recursion" — and is not restricted to the layered, feedforward structure of a multilayer perceptron. Backpropagation, as originally and still most commonly described, is reverse-mode AD applied to that one specific, more restricted computational shape (a network's layers composed with a loss). Every backpropagation computation is a reverse-mode AD computation; not every reverse-mode AD computation is what would historically have been called "backpropagation."

**Provenance:**
- source_url: https://jmlr.org/papers/v18/17-468.html (arXiv: https://arxiv.org/abs/1502.05767, Abstract)
- source_author: Baydin, Pearlmutter, Radul, Siskind
- source_date: 2018
- source_tier: 1
- exact quote: "Automatic differentiation (AD), also called algorithmic differentiation or simply 'autodiff', is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs."
- related existing notes: [[backpropagation-gap]], [[claim-linnainmaa-field-numerical-analysis]]

---

## Further leads

- Griewank & Walther's *Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation* (2nd ed., SIAM, 2008) formulates the "cheap gradient principle" with a specific numeric bound on reverse-mode cost relative to the original function's cost — not located/verified this session; would resolve the `[unverified-quant]` gap on the "small constant factor" claim above. `[unverified-quant — needs primary]`
- Andreas Griewank, "Who Invented the Reverse Mode of Differentiation?" *Documenta Mathematica*, Extra Volume ISMP (2012), pp. 389–400 — the dedicated history-of-mathematics paper already flagged as unread in [[claim-linnainmaa-reverse-mode-single-pass]] and [[claim-linnainmaa-priority-not-paternity]]; would directly verify the Linnainmaa-to-reverse-mode-AD identity claim that those notes currently carry as `[unverified-mechanism]`.
- Forward-mode AD (the dual technique, computing Jacobian-vector products via dual numbers, efficient when outputs ≫ inputs) is the mirror case to reverse mode and is described in the same Baydin et al. survey, Section 3.1 — not worked up here since it is not part of the backprop correspondence specifically.
- Colah's blog post "Calculus on Computational Graphs: Backpropagation" (https://colah.github.io/posts/2015-08-Backprop/) gives an accessible, widely-cited visual walkthrough of the computational-graph chain-rule mechanism — Tier 3-4 (independent blog), useful for exposition/pedagogy leads, not as a mechanism source given the Tier 1 source already secured here.
- Reverse-mode AD's "tape" or "Wengert list" (the recorded sequence of elementary operations from the forward pass that the backward pass replays) is the data structure underlying modern autograd systems (PyTorch, JAX, TensorFlow) — mentioned in passing in Baydin et al. but not worked up as its own claim here; a good candidate for a dedicated note connecting historical reverse-mode AD to current ML framework implementation.
- The Linnainmaa cluster ([[claim-linnainmaa-reverse-mode-single-pass]], [[claim-linnainmaa-thesis-identity]], [[claim-linnainmaa-field-numerical-analysis]], [[claim-linnainmaa-rounding-error-problem]], [[claim-linnainmaa-priority-not-paternity]]) already carries the historical half of this correspondence (who derived it first, in what field); this capture supplies the precise mechanism/definitional half from a Tier 1 AD-specific source rather than relying on Wikipedia alone.