Amari (1998) established that the ordinary gradient is not the steepest-descent direction when parameter space is non-Euclidean — the natural gradient, via the Fisher information metric, is

In "Natural Gradient Works Efficiently in Learning" (Neural Computation 10(2):251–276, 1998), Shun'ichi Amari showed that the familiar Euclidean gradient is the direction of steepest descent only under the implicit assumption that the parameter space is flat. In his words: "When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does."

The "natural" gradient is the ordinary gradient premultiplied by the inverse of the Fisher information matrix — the Riemannian metric tensor on the statistical manifold of the model's parameters. Where the ordinary gradient asks "which way increases the loss fastest per unit of coordinate change," the natural gradient asks "per unit of change in the distribution the parameters represent," which is the geometrically invariant question. Amari further argued the natural-gradient update is asymptotically Fisher-efficient — it attains the best possible asymptotic estimation accuracy. The paper is the best-known machine-learning payoff of information geometry, the field Amari founded, which applies Riemannian and differential-geometric tools (manifolds, the Fisher metric) to statistics and learning.

This is Amari's real, uncontested landmark — a useful complement to the vault's Amari myth cluster, which knocks down the false attribution that he trained multilayer perceptrons by SGD in 1967 (myth-amari-first-sgd-mlp, claim-wikipedia-amari-sgd-citogenesis) and documents his genuine associative-memory priority (claim-amari-1972-associative-memory-precedes-hopfield). It also sits at the "gradient is not flat Euclidean" pole shared by the backprop failure mode in claim-vanishing-gradient-chain-rule-pathology and, 28 years later and without citation, by the quantization-layer reading in claim-gift-2026-gradient-anisotropy-isotropic-transform. See also moc-backpropagation-origins.