Fisher information matrix

The local metric on a model's parameter manifold — the curvature object that says how much a distribution shifts per unit change in parameters. In machine learning it is the spine of the vault's gradient-geometry cluster: premultiplying the ordinary gradient by its inverse gives Amari's natural gradient (true steepest descent on the statistical manifold), and its tractable Kronecker-factored approximation (K-FAC, Martens & Grosse 2015) is what makes it usable at scale. A recurring subtlety, load-bearing enough to have its own open question: the empirical Fisher (gradient outer product from real labels) is not in general the same matrix as the true Fisher (from the model's sampled predictive distribution), a distinction Kunstner et al. (2019) argue is routinely blurred.

It is the specific object identified as shared between GIFT's gradient communication transform and Amari's natural gradient — and, by contrast, absent as a formal basis in GRADE's stable-rank construction.

References

• claim-amari-1998-natural-gradient-fisher-steepest-descent — the metric that defines the natural gradient
• claim-gift-isotropy-transform-derived-from-fisher-kfac — GIFT whitens gradients with the Fisher/K-FAC object
• claim-gift-restricts-fisher-kfac-object-to-communication-coordinates — GIFT uses it only for communication coordinates
• claim-grade-stable-rank-not-reduced-to-fisher — GRADE cites Fisher as inspiration, not derivation
• observation-gradient-geometry-shared-object-across-two-of-three — the three-way verdict
• claim-kunstner-empirical-fisher-not-equivalent-to-true-fisher — formal true-vs-empirical Fisher definitions and non-equivalence
• claim-martens-grosse-kfac-defines-true-fisher-convention — canonical K-FAC paper's Fisher is the true-Fisher convention
• claim-gift-g-factor-matches-empirical-fisher-not-true-fisher-convention — GIFT's own G factor reads as the empirical Fisher, answering the open thread below
• Resolved thread: question-gift-fisher-factor-true-or-empirical (answered 2026-07-27) · question-gradient-geometry-one-object-or-three-analogies
• Related hubs: entity-shunichi-amari · entity-k-fac