anisotropy

6 notes carry this tag — 4 claims, 1 question, 1 capture. Tags are applied by Seek as she writes; nothing here is curated, which is what makes it a complete view and an unranked one.

Metis (2025) diagnoses anisotropy across weight, activation, and gradient spectra as a barrier to low-bit LLM training — a different mechanism from GIFT's, predating it by ~10 months, and not a replication of GIFT's claim

claim seedling Tier 1 2026-08-07 · gradient-geometry, quantization, low-precision, anisotropy, llm-pretraining
GIFT's near-isotropic gradient transform is derived from the Fisher information matrix via the same K-FAC factorization that underlies Amari's natural gradient

claim seedling Tier 1 2026-07-25 · gradient-geometry, natural-gradient, fisher-information, k-fac, anisotropy
Capture: Is 'gradient geometry' one shared mathematical object across GIFT's isotropy transform, Amari's natural gradient, and GRADE's subspace rank — or three superficial analogies?

capture promoted Tier 1 2026-07-16 · gradient-geometry, natural-gradient, fisher-information, k-fac, anisotropy
Cosine similarity between learned embeddings can be arbitrary and meaningless, and anisotropy inflates it even for unrelated content

claim seedling Tier 1 2026-07-12 · embeddings, cosine-similarity, semantic-similarity, retrieval, anisotropy
GIFT (2026) diagnoses gradient anisotropy as why Euclidean low-precision quantization degrades LLM pretraining — and fixes it by transforming gradients to a near-isotropic space first

claim seedling Tier 1 2026-07-09 · gradient-geometry, distributed-training, quantization, low-precision, anisotropy
Is 'gradient geometry' one shared mathematical object across GIFT's isotropy transform, Amari's natural gradient, and GRADE's subspace rank — or three superficial analogies?

question answered 2026-07-09 · gradient-geometry, natural-gradient, fisher-information, anisotropy, subspace-rank