Capture: Is GIFT's K-FAC gradient factor G = E[δδ⊤] the true Fisher (model's sampled predictive distribution) or the empirical Fisher (real training labels) — and does the difference weaken the GIFT–Amari 'same object' bridge?

This capture directly resolves the hook left open in question-gift-fisher-factor-true-or-empirical (raised 2026-07-25 while promoting claim-gift-isotropy-transform-derived-from-fisher-kfac): whether GIFT's G = E[δδ⊤] K-FAC factor is the true Fisher (expectation over the model's own sampled predictive distribution, as Amari's construction and the canonical K-FAC paper assume) or the empirical Fisher (built from real training labels, per Kunstner et al.'s critique). All three PDFs were fetched fresh this session via extract_pdf; all three returned tls: "verified". See Safety flags below (none fired).

Claim: Kunstner, Balles & Hennig (2019) formally define the true Fisher and the "empirical Fisher" as two different matrices, and show the empirical Fisher is not, in general, a valid estimate of the true Fisher

Claim type: definitional / technical-mechanism. Floor: Tier 1–2 required. Achieved: Tier 1 (Kunstner et al., arXiv:1905.12558, read directly).

The paper defines the Fisher information matrix as an expectation taken over the model's own predictive distribution: "F(θ) := Σₙ E_{p_θ(y|xₙ)}[∇_θ log p_θ(y|xₙ) ∇_θ log p_θ(y|xₙ)ᵀ]" and the widely-used approximation it calls the empirical Fisher as "F̃(θ) := Σₙ ∇_θ log p_θ(yₙ|xₙ) ∇_θ log p_θ(yₙ|xₙ)ᵀ," where yₙ is the actual training label rather than a value drawn from the model. The paper is explicit that swapping one for the other is not a harmless approximation: "yₙ is a training label and not a sample from the model's predictive distribution p_θ(y|xₙ). Therefore, and contrary to what its name suggests, the empirical Fisher is not an empirical (i.e. Monte Carlo) estimate of the Fisher." It further states that convergence between the two "depends on how close the model p_θ(y|xₙ) is to the true data-generating distribution p(y|xₙ)" — i.e. they coincide only near a well-fit optimum, not in general during training. The paper's central argument is that the conditions under which the empirical Fisher approaches the true Fisher "are unlikely to be met in practice," and that using it "can lead to highly undesirable effects" as a curvature approximation.

This is the formal vocabulary the rest of this capture depends on: "true Fisher" (model-sampled) and "empirical Fisher" (real-label) name two matrices that are not interchangeable by definition, and are not guaranteed to be close in practice.

Claim: The canonical K-FAC paper (Martens & Grosse, 2015) — which both GIFT and Kunstner et al. build on — defines its Fisher using the model's own predictive distribution over outputs, i.e. the true-Fisher convention, not real training labels

Claim type: technical-mechanism. Floor: Tier 1–2 required. Achieved: Tier 1 (Martens & Grosse, arXiv:1503.05671, read directly).

Martens & Grosse define the quantity K-FAC approximates as "F = E[DθDθᵀ]" and specify exactly which expectation this is: "Here, the expectation is taken with respect to the data distribution Q̂ₓ over inputs x, and the model's predictive distribution P_{y|x}(θ) over y." That is the true-Fisher convention in Kunstner et al.'s terminology (expectation over model-sampled y, not real labels) — the paper that gave K-FAC its name defines the object K-FAC approximates as requiring y sampled from the network's own output distribution, not from the training set. This matches Kunstner et al.'s own aside that "KFAC [Martens and Grosse, 2015]... [has] been re-implemented by third parties using the empirical Fisher" — implying the original K-FAC formulation used the true Fisher, and that empirical-Fisher K-FAC variants are a later, distinct substitution made by other authors.

Claim: GIFT's own method section defines its δ (and hence G = E[δδ⊤]) as the ordinary backpropagated gradient computed during real pretraining, never states that y is resampled from the model's predictive distribution, and never uses the terms "true Fisher" or "empirical Fisher"

Claim type: technical-mechanism. Floor: Tier 1–2 required. Achieved: Tier 1 (GIFT, arXiv:2607.07494, method section read directly). Caveat: this is an inference from what GIFT's text does and does not say, not a label GIFT applies to itself — flagged explicitly below.

GIFT's background section defines the quantity directly: "let a denote the layer input activation and let δ denote the gradient with respect to the layer pre-activation output. Using the standard column-major vectorization convention, K-FAC approximates the layerwise Fisher block as F_W ≈ A ⊗ G, A = E[aa⊤], G = E[δδ⊤]." Nowhere in the paper's background, design, or implementation sections is there any mention of sampling y from the model's own predictive distribution, Monte Carlo estimation of the Fisher, or the terms "true Fisher" / "empirical Fisher" (confirmed by a full read of the paper's 12 pages, including the implementation section describing how the factors are estimated online "every 50 training steps" during actual Llama pretraining on OpenWebText). The δ used is, by the paper's own account, the real backpropagated gradient produced by the actual training loss against actual training targets during ordinary pretraining — not a separately-sampled quantity.

Structurally, this matches Kunstner et al.'s definition of the empirical Fisher (built from actual labels yₙ) rather than the true-Fisher convention that both Amari's natural-gradient construction and Martens & Grosse's original K-FAC paper specify (expectation over model-sampled y). GIFT itself does not classify which Fisher its G approximates — the paper simply never raises the distinction — so this claim rests on comparing GIFT's operational definition of δ against the two primary definitions above, not on an explicit statement in GIFT's own text.

Claim: On this reading, GIFT's G factor is the empirical Fisher rather than the true Fisher its cited K-FAC ancestor and Amari's construction assume — which weakens the GIFT–Amari "same object" identity from a strict mathematical identity to "same factorization scheme, applied to two different Fisher variants"

Claim type: technical-mechanism (synthesis of the three claims above). Floor: Tier 1–2 required. Achieved: Tier 1 (built directly from the three Tier-1 primary sources above; no new source needed for the inference itself).

Putting the three findings together: Amari's natural gradient and Martens & Grosse's canonical K-FAC are both formally defined over the true Fisher (expectation over the model's own sampled predictive distribution); GIFT's G = E[δδ⊤], by its own operational definition, is built from ordinary real-label backpropagated gradients during real pretraining, which is structurally the empirical Fisher; and Kunstner et al. show the empirical Fisher is not, in general, the same matrix as the true Fisher, nor a valid estimate of it, except under conditions (near-perfect model fit to the data) unlikely to hold during LLM pretraining. If this reading is correct, claim-gift-isotropy-transform-derived-from-fisher-kfac's framing — that GIFT and Amari's natural gradient are "two operations built on the same formal object" — needs a qualifier: they use the same Kronecker-factorization scheme (F ≈ A ⊗ G), but the G factor each is formally defined over is not the same matrix in the strict sense Kunstner et al. establish. The bridge survives as "same K-FAC machinery, applied to two different Fisher variants that are not generally interchangeable," rather than as an unqualified identity.

Central question status

[largely resolved, with one inferential step flagged] — GIFT's own method section defines its G = E[δδ⊤] factor using the real backpropagated training gradient (δ), with no mention of resampling outputs from the model's predictive distribution anywhere in the paper; this matches Kunstner et al.'s formal definition of the empirical Fisher, not the true Fisher that both Amari's 1998 construction and Martens & Grosse's 2015 canonical K-FAC paper explicitly define via an expectation over the model's own sampled outputs. Because Kunstner et al. show the two are not generally equivalent, the difference does weaken the strict "same object" reading of the GIFT–Amari bridge — though the weakening is qualified, not fatal: both papers still use the identical Kronecker block-factorization scheme (F ≈ A ⊗ G), and GIFT itself never explicitly labels which Fisher it computes, so the true/empirical classification here is a reasoned inference from GIFT's silence on the sampling question, not a claim GIFT makes about itself. A fully closed resolution would require either an explicit statement from GIFT's authors, or a derivation showing the specific conditions under which their real-label G behaves like the true Fisher in this setting — neither of which exists in the current source set.

Safety flags

None. All three PDF fetches this session (GIFT: arXiv:2607.07494, Kunstner et al.: arXiv:1905.12558, Martens & Grosse: arXiv:1503.05671) returned tls: "verified" via extract_pdf, and no page showed addressed-to-AI language, override language, claimed authority, tier self-assignment, file-system instructions, credential requests, or urgency framing.

Further leads

• Kunstner et al. also argue (§4–5) that empirical-Fisher-based methods' practical success (e.g. Adam) is better explained as adaptation to gradient noise rather than to curvature — a distinct thread from GIFT, worth a separate note if the vault wants to trace why empirical-Fisher approximations work despite the theoretical gap.
• Martens & Grosse's paper notes their own inverse-Fisher approximation is justified as block-diagonal/tridiagonal "even though this justification doesn't apply to the [forward] Fisher itself" — a separate approximation-layer subtlety inside K-FAC not explored here.
• Whether GIFT's choice of real-label δ (vs. model-sampled) was a deliberate engineering simplification or simply unexamined was not addressed by any source found this session; no GIFT author commentary, blog post, or errata was located.
• GRADE's stable-rank construction (claim-grade-stable-rank-not-reduced-to-fisher) remains formally unrelated to either Fisher variant discussed here — not re-litigated in this capture.

Entity candidates

• Frederik Kunstner — person — lead author of the 2019 true-vs-empirical-Fisher critique this capture leans on; not yet represented in the vault as its own entity.
• James Martens — person — co-author of the canonical K-FAC paper (2015); primary source now directly read for the true-Fisher convention it specifies.
• Roger Grosse — person — co-author of the canonical K-FAC paper (2015).
• empirical Fisher (vs. true Fisher) — concept — the specific formal distinction this capture turns on; recurring enough across the vault's gradient-geometry cluster (entity-fisher-information-matrix, entity-k-fac) to warrant its own concept note rather than living only inside aliases.
• generalized Gauss-Newton (GGN) — concept — the matrix Kunstner et al. use to formally characterize when the empirical Fisher does and doesn't behave like a valid curvature approximation; appears load-bearing in their argument but not explored in this capture.