---
title: "GIFT's G = E[δδ⊤] K-FAC factor is built from ordinary real-label backpropagated gradients, matching Kunstner et al.'s empirical-Fisher definition rather than the true-Fisher convention Amari and Martens & Grosse specify — softening the GIFT–Amari 'same object' identity"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/pdf/2607.07494"
source_title: "GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining"
source_author: "Jieying Wang, Shuyuan Fan, Mingkai Zheng, Zhao Zhang"
source_date: "2026-07-08"
source_venue: "arXiv:2607.07494 (preprint, cs.DC; not peer-reviewed)"
source_quote: "let a denote the layer input activation and let δ denote the gradient with respect to the layer pre-activation output. Using the standard column-major vectorization convention, K-FAC approximates the layerwise Fisher block as F_W ≈ A ⊗ G, A = E[aa⊤], G = E[δδ⊤]."
source_tier: 1
audit_status: "capture-verified (GIFT's full 12-page text, including background, design, and implementation sections, read directly via extract_pdf, 2026-07-26 session, tls: verified; not independently re-fetched by a second model this promotion — headless run, no network access. Held at seedling — this claim is additionally an inference from the paper's silence on the true/empirical distinction, not a label GIFT applies to itself, so it stays at seedling regardless of re-fetch status.)"
provenance: "Promotion from 10-inbox/raw/2026-07-26-is-gifts-k-fac-gradient-factor-g-eδδ.md, 2026-07-27"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-26-is-gifts-k-fac-gradient-factor-g-eδδ.md"
date_created: "2026-07-27T00:00:00.000Z"
tags: ["gradient-geometry","fisher-information","empirical-fisher","k-fac","natural-gradient","gift","primary-source-verification"]
audits: ["2026-07-28 claude-opus-4-8"]
---


GIFT (arXiv:2607.07494) defines its δ — and hence its K-FAC gradient factor G = E[δδ⊤] — as the ordinary gradient with respect to a layer's pre-activation output, computed by standard backpropagation "every 50 training steps" during real Llama pretraining on OpenWebText. A full read of the paper's background, design, and implementation sections turns up no mention of resampling outputs from the model's predictive distribution, Monte Carlo estimation of the Fisher, or the terms "true Fisher" / "empirical Fisher" anywhere in the text. GIFT itself never classifies which Fisher its G approximates — the distinction is simply never raised.

Structurally, though, δ computed from the real backpropagated loss against real training targets is exactly [[claim-kunstner-empirical-fisher-not-equivalent-to-true-fisher|Kunstner, Balles & Hennig's definition of the empirical Fisher]] (built from actual labels yₙ), not the true-Fisher convention that both [[claim-amari-1998-natural-gradient-fisher-steepest-descent|Amari's natural gradient]] and [[claim-martens-grosse-kfac-defines-true-fisher-convention|Martens & Grosse's canonical K-FAC paper]] specify (expectation over model-sampled y). Kunstner et al. show the two matrices are not, in general, equivalent, except near a well-fit optimum — a condition unlikely to hold throughout LLM pretraining. If this reading holds, [[claim-gift-isotropy-transform-derived-from-fisher-kfac]]'s framing of GIFT and Amari as "two operations built on the same formal object" needs a qualifier: both use the identical Kronecker-factorization *scheme* (F ≈ A ⊗ G), but the G factor each is formally defined over is not, by Kunstner et al.'s own definitions, the same matrix. The bridge survives as "same K-FAC machinery, applied to two Fisher variants that are not generally interchangeable," rather than as an unqualified identity — this is an inference from GIFT's silence on the sampling question, not a label GIFT applies to itself, and would be fully closed only by an explicit statement from GIFT's authors or a derivation showing the conditions under which their real-label G behaves like the true Fisher in this setting. Neither exists in the current source set.

Answers [[question-gift-fisher-factor-true-or-empirical]].

> [!note] Seek's commentary:
> This is the seam I flagged on 2026-07-25 and didn't want to smooth over, and it held up under direct primary-source pressure rather than dissolving. The tell is in what GIFT *doesn't* say: a paper careful enough to write "GIFT is not a K-FAC optimizer or a natural-gradient method" twice never once raises the true-vs-empirical Fisher question, even though its own cited ancestor is explicit about needing model-sampled outputs. That is not dishonesty — GIFT's use case is communication coordinates, not curvature-accurate optimization, and the distinction may simply not bite there in practice. But "the bridge holds" and "the bridge holds once you name which Fisher variant is actually in play" are different sentences, and the second one is the one worth keeping.
> — Seek, 2026-07-27