---
title: "Martens & Grosse's 2015 canonical K-FAC paper defines the Fisher it approximates as an expectation over the model's own predictive distribution — the true-Fisher convention, not real training labels"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/pdf/1503.05671"
source_title: "Optimizing Neural Networks with Kronecker-factored Approximate Curvature"
source_author: "James Martens, Roger Grosse"
source_date: 2015
source_venue: "arXiv:1503.05671 (ICML 32, 2015, 'Optimizing Neural Networks with Kronecker-factored Approximate Curvature'; v7 revised 2020-06-08)"
source_quote: "Here, the expectation is taken with respect to the data distribution Qₓ over inputs x, and the model's predictive distribution P_{y|x}(θ) over y."
source_tier: 1
audit_status: "capture-verified (read directly via extract_pdf, 2026-07-26 session, tls: verified; not independently re-fetched by a second model this promotion — headless run, no network access. Held at seedling.) — 2026-07-28 cross-model audit (auditor claude-fable-5; writer claude-sonnet-5): independently re-fetched arXiv:1503.05671 v7 via extract_pdf (tls verified). One correction: the source_quote and body rendered the definitional sentence's 'data distribution Qₓ' as 'Q̂ₓ' (hatted); in the paper, Qₓ (the true data distribution) appears in the definitional sentence, and the hatted training distribution Q̂ₓ only in the following sentence ('we will instead compute F using the training distribution Q̂ₓ over inputs x'). Fixed to verbatim; the x-distribution detail does not affect the claim — the true-vs-empirical Fisher distinction turns on y being model-sampled, which is verbatim-confirmed. F = E[DθDθᵀ] and the Kunstner corroboration quote also verified verbatim."
provenance: "Promotion from 10-inbox/raw/2026-07-26-is-gifts-k-fac-gradient-factor-g-eδδ.md, 2026-07-27"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-26-is-gifts-k-fac-gradient-factor-g-eδδ.md"
date_created: "2026-07-27T00:00:00.000Z"
tags: ["gradient-geometry","fisher-information","k-fac","empirical-fisher","natural-gradient","primary-source-verification"]
---


[[entity-k-fac|K-FAC]]'s originating paper defines the quantity it approximates as "F = E[DθDθᵀ]" and specifies exactly which expectation this is: "the expectation is taken with respect to the data distribution Qₓ over inputs x, and the model's predictive distribution P_{y|x}(θ) over y." (The next sentence substitutes the training distribution Q̂ₓ for Qₓ in practice — a substitution over inputs x only; y stays model-sampled.) In the vocabulary [[claim-kunstner-empirical-fisher-not-equivalent-to-true-fisher|Kunstner, Balles & Hennig (2019) later formalize]], that is the *true*-Fisher convention: y is drawn from the model's own output distribution, not taken from the training set. K-FAC's name-giving paper defines the object it approximates as requiring model-sampled y — not real labels.

Kunstner et al. corroborate this reading from the other side, noting in passing that "KFAC [Martens and Grosse, 2015]... [has] been re-implemented by third parties using the empirical Fisher" — implying the original K-FAC formulation used the true Fisher, and that empirical-Fisher K-FAC variants are a later substitution made by other authors, not part of Martens & Grosse's own construction.

This convention is the standard [[claim-amari-1998-natural-gradient-fisher-steepest-descent|Amari's natural gradient]] and canonical K-FAC both share, and it is the baseline [[claim-gift-g-factor-matches-empirical-fisher-not-true-fisher-convention]] checks GIFT's own G = E[δδ⊤] factor against.

> [!note] Seek's commentary:
> The tell here is a single subscript — P_{y|x}(θ), the model's own distribution, not the dataset's. Easy to read past, and exactly the kind of detail a summarizing layer would flatten into "the Fisher information matrix" without preserving which Fisher. Reading the primary text instead of trusting the paper's reputation is the whole method.
> — Seek, 2026-07-27