gradient-geometry

21 notes carry this tag — 11 claims, 2 observations, 3 questions, 5 captures. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

Does GIFT's claimed 7.6% Llama-600M pretraining speedup (and its anisotropy-distortion diagnosis) hold up under peer review or independent replication?

capture promoted 2026-08-07 · gradient-geometry, quantization, distributed-training, quant-verification, primary-source-verification
As of 2026-08-07, GIFT's 7.6% Llama-600M speedup remains an unrefereed, unreplicated preprint figure

claim seedling Tier 1 2026-08-07 · gradient-geometry, quantization, distributed-training, gift, peer-review-status
Metis (2025) diagnoses anisotropy across weight, activation, and gradient spectra as a barrier to low-bit LLM training — a different mechanism from GIFT's, predating it by ~10 months, and not a replication of GIFT's claim

claim seedling Tier 1 2026-08-07 · gradient-geometry, quantization, low-precision, anisotropy, llm-pretraining
GIFT's G = E[δδ⊤] K-FAC factor is built from ordinary real-label backpropagated gradients, matching Kunstner et al.'s empirical-Fisher definition rather than the true-Fisher convention Amari and Martens & Grosse specify — softening the GIFT–Amari 'same object' identity

claim seedling Tier 1 2026-07-27 · gradient-geometry, fisher-information, empirical-fisher, k-fac, natural-gradient
Kunstner, Balles & Hennig (2019) formally distinguish the true Fisher from the empirical Fisher and show the empirical Fisher is not, in general, a valid estimate of it

claim seedling Tier 1 2026-07-27 · gradient-geometry, fisher-information, empirical-fisher, natural-gradient, primary-source-verification
Martens & Grosse's 2015 canonical K-FAC paper defines the Fisher it approximates as an expectation over the model's own predictive distribution — the true-Fisher convention, not real training labels

claim seedling Tier 1 2026-07-27 · gradient-geometry, fisher-information, k-fac, empirical-fisher, natural-gradient
Capture: Is GIFT's K-FAC gradient factor G = E[δδ⊤] the true Fisher (model's sampled predictive distribution) or the empirical Fisher (real training labels) — and does the difference weaken the GIFT–Amari 'same object' bridge?

capture promoted Tier 1 2026-07-26 · gradient-geometry, fisher-information, empirical-fisher, k-fac, natural-gradient
GIFT's near-isotropic gradient transform is derived from the Fisher information matrix via the same K-FAC factorization that underlies Amari's natural gradient

claim seedling Tier 1 2026-07-25 · gradient-geometry, natural-gradient, fisher-information, k-fac, anisotropy
GIFT uses the Fisher/K-FAC object only to define communication coordinates and explicitly, repeatedly denies being a natural-gradient or K-FAC optimizer

claim seedling Tier 1 2026-07-25 · gradient-geometry, natural-gradient, fisher-information, k-fac, quantization
GRADE's gradient-subspace stable rank is a normalized covariance-spectrum construction with no established formal reduction to the Fisher information matrix

claim seedling Tier 1 2026-07-25 · gradient-geometry, fisher-information, stable-rank, subspace-rank, grade
Gradient 'geometry' is one shared object — the Fisher/K-FAC matrix — across GIFT and Amari, but a formally distinct construction in GRADE: not one object across all three, nor three unrelated analogies

observation seedling Tier 1 2026-07-25 · gradient-geometry, fisher-information, k-fac, natural-gradient, subspace-rank
Is GIFT's K-FAC gradient factor G = E[δδ⊤] the true Fisher (model's sampled predictive distribution) or the empirical Fisher (real training labels) — and does the difference weaken the GIFT–Amari 'same object' bridge?

question answered 2026-07-25 · gradient-geometry, fisher-information, empirical-fisher, k-fac, natural-gradient
Capture: Is 'gradient geometry' one shared mathematical object across GIFT's isotropy transform, Amari's natural gradient, and GRADE's subspace rank — or three superficial analogies?

capture promoted Tier 1 2026-07-16 · gradient-geometry, natural-gradient, fisher-information, k-fac, anisotropy
Perceptual colour space is not even Riemannian, because diminishing returns cannot exist in a Riemannian geometry (Bujack 2022)

claim seedling Tier 1 2026-07-12 · color-science, perceptual-color, riemannian-geometry, non-euclidean, information-geometry
CIELAB's non-Euclidean-metric patch is a fourth instance of the gradient-geometry pattern (Amari, GIFT) — not a kin of the hub-selection sampling artifact

observation seedling Tier 1 2026-07-12 · color-science, cielab, non-euclidean, information-geometry, natural-gradient
CIELAB's blue-hue failure bridges to natural-gradient geometry, not to the hub-selection artifact

capture promoted Tier 1 2026-07-11 · color-science, cielab, non-euclidean, information-geometry, natural-gradient
GIFT's gradient-anisotropy fix echoes Amari's 1998 natural gradient

capture promoted 2026-07-09 · hop, gradient-geometry, natural-gradient, distributed-training, amari
Amari (1998) established that the ordinary gradient is not the steepest-descent direction when parameter space is non-Euclidean — the natural gradient, via the Fisher information metric, is

claim seedling Tier 1 2026-07-09 · gradient-geometry, natural-gradient, information-geometry, fisher-information, amari
GIFT (2026) diagnoses gradient anisotropy as why Euclidean low-precision quantization degrades LLM pretraining — and fixes it by transforming gradients to a near-isotropic space first

claim seedling Tier 1 2026-07-09 · gradient-geometry, distributed-training, quantization, low-precision, anisotropy
Is 'gradient geometry' one shared mathematical object across GIFT's isotropy transform, Amari's natural gradient, and GRADE's subspace rank — or three superficial analogies?

question answered 2026-07-09 · gradient-geometry, natural-gradient, fisher-information, anisotropy, subspace-rank
Does GIFT's claimed 7.6% Llama-600M pretraining speedup (and its anisotropy-distortion diagnosis) hold up under peer review or independent replication?

question open 2026-07-09 · gradient-geometry, quantization, distributed-training, quant-verification, primary-source-verification