Capture: How does forward-mode AD via dual numbers compute Jacobian-vector products, and what is its structural relationship to reverse-mode AD?

Core question: how does forward-mode AD via dual numbers compute Jacobian-vector products, and what is its structural relationship to reverse-mode AD?

Claim type: technical-mechanism Sourcing floor: Tier 1–2 required — met via Baydin et al. 2018 (arXiv/JMLR survey, Tier 1) and JAX's official documentation (Tier 1, primary self-documentation).

Finding: both halves of the question resolve cleanly against Tier 1 sources. Forward-mode AD computes a Jacobian-vector product (JVP) by propagating a "tangent" alongside each intermediate value of the computation, using dual-number arithmetic (or an equivalent program transformation) to carry the derivative information; reverse-mode AD computes the structurally dual object — a vector-Jacobian product (VJP) — via a two-phase forward/backward sweep, and the two are related as transposes of each other's linear map (equivalently, as pushforward and pullback of the same function). Details and exact sourcing below, organized by sub-claim.

Claim 1: Dual numbers are defined as truncated Taylor series with a nilpotent unit, and forward-mode AD is mathematically equivalent to evaluating a function over dual numbers

Claim type: technical-mechanism + definitional Sourcing floor: Tier 1–2 required for the mechanism — met (Tier 1, primary arXiv survey text)

Baydin et al. define dual numbers precisely, in Section 3.1.1:

"truncated Taylor series of the form v+v˙ϵ, where v,v˙∈ℝ and ϵ is a nilpotent number such that ϵ²=0 and ϵ≠0"

The arithmetic rule that makes this carry derivative information through multiplication mirrors the product rule:

"(v+v˙ϵ)(u+u˙ϵ)=(vu)+(vu˙+v˙u)ϵ"

And the paper states the mechanism's core equivalence directly, in Section 3.1:

"Mathematically, forward mode AD (represented by the left- and right-hand sides in Table 2) can be viewed as evaluating a function using dual numbers"

The same section notes there is more than one way to implement this equivalence — not only literal dual-number data types:

"This can be implemented through calls to a specific library, in the form of source code transformation where a given source code will be automatically modified, or through operator overloading, making the process transparent to the user."

This last point matters for the "how" of the question: dual-number arithmetic is the mathematical model of forward-mode AD, but concrete implementations may realize it either as literal dual-number objects (operator overloading) or as an equivalent source-transformed/traced program (see Claim 4 below on JAX, which does not use a literal dual-number type but implements the mathematically equivalent JVP transformation).

Related notes: dual-numbers, forward-mode-automatic-differentiation, jacobian-vector-product

Claim 2: Forward-mode AD computes a directional derivative / JVP by seeding the input tangent and propagating it forward through the computation

Claim type: technical-mechanism Sourcing floor: Tier 1–2 required — met (Tier 1, primary arXiv survey text)

Baydin et al. describe how a specific directional derivative is obtained by choosing the seed vector for the tangent component before running the (dual-number-valued) computation:

"we can obtain the directional derivative along a given vector r as a linear combination of the partial derivatives ∇f·r by starting the AD computation with the values ẋ=r"

JAX's own documentation states the equivalent, more general JVP framing directly — defining the JVP as a map and describing its per-primitive execution mechanism:

"Given a function f : ℝⁿ → ℝᵐ, the Jacobian-vector product maps (x, v) to ∂f(x) v"

"For each primitive numerical operation that the original function would have applied, the jvp-transformed function executes a 'JVP rule' for that primitive that both evaluates the primitive on the primals and applies the primitive's JVP at those primal values."

— JAX documentation, "Forward- and reverse-mode autodiff in JAX," https://docs.jax.dev/en/latest/jacobian-vector-products.html (Tier 1; no publication date displayed on the page, accessed 2026-07-06).

Related notes: jacobian-vector-product, forward-mode-automatic-differentiation

Claim 3: Forward-mode AD's efficiency scales with the number of inputs; a single forward pass yields all output derivatives for f: ℝ → ℝᵐ, but a full gradient of f: ℝⁿ → ℝ requires n passes

Claim type: technical-mechanism + quantitative (comparative cost) Sourcing floor: Tier 1–2 required — met (Tier 1, primary arXiv survey text)

"Forward mode AD is efficient and straightforward for functions f:ℝ→ℝᵐ, as all the derivatives dyᵢ/dx can be computed with just one forward pass."

By contrast, for the gradient case:

"forward mode AD requires n evaluations to compute the gradient ∇f=(∂y/∂x₁,…,∂y/∂xₙ)"

— and the survey concludes this motivates reverse mode for the opposite shape: "for cases f:ℝⁿ→ℝᵐ where n≫m, a different technique is often preferred" (i.e., reverse mode).

JAX's documentation states the same tradeoff in matrix-construction terms:

"Forward-mode builds Jacobian matrices one column at a time" while "reverse-mode builds Jacobian matrices one row at a time."

"Forward-mode is more efficient for 'tall' Jacobian matrices (more outputs than inputs), while reverse-mode is more efficient for 'wide' Jacobian matrices (more inputs than outputs)."

— JAX documentation, "Forward- and reverse-mode autodiff in JAX," https://docs.jax.dev/en/latest/jacobian-vector-products.html (Tier 1).

Note the "n evaluations" and "one column/row at a time" figures are structural/asymptotic claims about algorithm shape, not empirically measured benchmark numbers — both sources present them as direct mathematical consequences of the method, not as measured data, which is why they can rest on the survey/documentation text itself rather than needing an independent benchmark citation.

Related notes: forward-mode-automatic-differentiation, reverse-mode-automatic-differentiation

Claim 4: Reverse-mode AD works as a two-phase forward/backward sweep, propagating "adjoints" backward through a recorded computational graph

Claim type: technical-mechanism Sourcing floor: Tier 1–2 required — met (Tier 1, primary arXiv survey text)

"In reverse mode AD, derivatives are computed in the second phase of a two-phase process. In the first phase, the original function code is run forward, populating intermediate variables vᵢ and recording the dependencies in the computational graph through a bookkeeping procedure. In the second phase, derivatives are calculated by propagating adjoints v̄ᵢ in reverse, from the outputs to the inputs."

The adjoint itself is defined as a sensitivity:

"This is done by complementing each intermediate variable vᵢ with an adjoint v̄ᵢ=∂yⱼ/∂vᵢ, which represents the sensitivity of a considered output yⱼ with respect to changes in vᵢ."

— Baydin et al. 2018, Section 3.2 (Reverse Mode).

Related notes: reverse-mode-automatic-differentiation, backpropagation

Claim 5: The structural relationship between forward- and reverse-mode AD — JVP and VJP as transposes of the same linear map, equivalently pushforward and pullback

Claim type: technical-mechanism Sourcing floor: Tier 1–2 required — met (Tier 1, JAX's own documentation)

This is the most precise statement of "structural relationship" found in this search. JAX's documentation defines VJP explicitly as the transpose of JVP's linear part:

"The linear part of a VJP as the transpose (or adjoint conjugate) of the linear part of a JVP: (x, v) ↦ ∂f(x)ᵀ v"

and gives the vector-Jacobian-product definition directly, parallel to the JVP definition in Claim 2:

"Given a function f : ℝⁿ → ℝᵐ, the vector-Jacobian product notation is (x, v) ↦ vᵀ∂f(x), where v is an element of the cotangent space"

The same documentation frames this in differential-geometric language — JVP as a pushforward, VJP as a pullback:

"This map is called the pushforward map of f at x."

"The corresponding map on cotangent spaces is often called the pullback of f at x."

"The key for our purposes is that it goes from something that looks like the output of f to something that looks like the input of f, just like we might expect from a transposed linear function."

— JAX documentation, "Forward- and reverse-mode autodiff in JAX," https://docs.jax.dev/en/latest/jacobian-vector-products.html (Tier 1).

This directly answers the "structural relationship" half of the harvested question: forward-mode (JVP) and reverse-mode (VJP) are not two unrelated algorithms but the same underlying linear map (the Jacobian of f at a point) applied in two dual directions — forward along tangent vectors (pushforward), reverse along cotangent/adjoint vectors (pullback), with VJP's linear action being literally the transpose of JVP's.

Related notes: jacobian-vector-product, vector-jacobian-product, forward-mode-automatic-differentiation, reverse-mode-automatic-differentiation, pushforward-pullback

Claim 6: Forward-mode JVP evaluation costs roughly 3x a plain function evaluation and uses memory independent of computation depth; reverse-mode VJP trades that for memory that scales with computation depth

Claim type: quantitative Sourcing floor: Tier 1–2 required — met (Tier 1, JAX's own documentation) — flagged as implementation-specific, not a universal AD-theory constant

"The FLOP cost of the jvp-transformed function is about 3x the cost of just evaluating the function...the memory cost is independent of the depth of the computation."

"Though the FLOPs are friendly [for reverse-mode], memory scales with the depth of the computation" while forward-mode has memory independent of computation depth.

— JAX documentation, "The Autodiff Cookbook," https://docs.jax.dev/en/latest/notebooks/autodiff_cookbook.html (Tier 1).

This "~3x" figure is JAX's own characterization of its own implementation's cost profile (one unit for the primal evaluation, one for linearizing, one for applying the linearized map to a vector), not a claim about every possible forward-mode AD implementation. It is recorded here as sourced to Tier 1 but scoped narrowly — it should not be generalized on promotion into a blanket "forward-mode AD always costs 3x" claim without that caveat.

Related notes: forward-mode-automatic-differentiation, reverse-mode-automatic-differentiation

What was not found / left as a gap

No Tier 1–2 source was located in this pass that traces the specific historical origin of using dual numbers specifically (as opposed to other forward-mode implementation strategies) for automatic differentiation — e.g., an attribution to a specific paper or author for "dual numbers as an AD technique" distinct from dual numbers' 19th-century origin in abstract algebra (Clifford). This was outside the scope of the harvested question (which asks about mechanism and structural relationship, not history), so it was not chased further, but is flagged as a lead if a future capture wants the historical thread.