K-FAC (Kronecker-Factored Approximate Curvature)

The standard tractable approximation to the Fisher information matrix, introduced by Martens & Grosse (2015). For a linear layer it factors the layer's Fisher block as a Kronecker product of two smaller matrices — F_W ≈ A ⊗ G, where A = E[aa⊤] captures input-side second-order statistics and G = E[δδ⊤] the output-gradient side — reducing an intractable full curvature matrix to per-layer factors that can be inverted cheaply.

It is the specific mechanism by which the GIFT–Amari bridge is built: GIFT's near-isotropic gradient transform and Amari-style natural-gradient methods both approximate the same Fisher curvature through this identical Kronecker factorization. GIFT is careful to note it uses the K-FAC block only as a local metric for communication coordinates, not as an optimizer.

References

• claim-gift-isotropy-transform-derived-from-fisher-kfac — GIFT's transform uses the K-FAC factorization F_W ≈ A ⊗ G
• claim-gift-restricts-fisher-kfac-object-to-communication-coordinates — "GIFT is not a K-FAC optimizer"
• observation-gradient-geometry-shared-object-across-two-of-three — K-FAC as the shared-object mechanism
• claim-martens-grosse-kfac-defines-true-fisher-convention — the originating paper's Fisher is the true-Fisher (model-sampled) convention
• claim-gift-g-factor-matches-empirical-fisher-not-true-fisher-convention — GIFT's own G = E[δδ⊤] factor reads as the empirical Fisher instead
• Resolved thread: question-gift-fisher-factor-true-or-empirical (answered 2026-07-27 — which Fisher the G = E[δδ⊤] factor approximates)
• Related hubs: entity-fisher-information-matrix · entity-shunichi-amari