Pearlmutter's 1994 R-operator computes an exact Hessian-vector product in O(n) time without ever forming the Hessian

automatic-differentiationhessian-vector-productbarak-pearlmutterreverse-mode-adsecond-order-methodsmechanism

Barak Pearlmutter's "Fast Exact Multiplication by the Hessian" (Neural Computation, 1994; written at Siemens Corporate Research, Princeton) gives a way to compute the product of a function's Hessian with an arbitrary vector, Hv, in time and space proportional to computing the gradient itself — O(n) — rather than the O(n²) cost of first building the full n×n Hessian. The trick is a differential operator, the R-operator, applied by nesting differentiation: a forward-mode directional derivative taken through the reverse-mode gradient computation. The result is exact, not a finite-difference approximation: "This allows information to be extracted from the Hessian without ever calculating or storing the Hessian itself."

The construction is an early, concrete instance of composing the two modes of automatic differentiation — forward-over-reverse — to obtain a second-order quantity cheaply. Because backpropagation is reverse-mode AD applied to a network-plus-loss, the Hessian-vector product is what makes curvature usable inside neural-network training without the quadratic blowup: it is the primitive underneath conjugate-gradient and Krylov-subspace second-order optimizers, Gauss-Newton and natural-gradient methods, and influence-function computations. The paper is by the same Pearlmutter who later co-authored the survey behind claim-ml-and-ad-communities-mutually-unaware — a result derived at an industrial lab a decade before the ML and AD literatures began citing each other in earnest.

The mechanism above is recorded from a direct read of the 1994 paper (Tier 1; source_sha carried). A separate assertion in the capture — that PyTorch and JAX documentation still call this construction "the Pearlmutter trick" by name in 2026 — rests on WebSearch corroboration only and is not verified here; it is carried as a watch_flag lead, not as a sourced claim, and does not bear on the mechanism claim in the title.