intrinsic dimension

The smallest number of free parameters — in a random low-dimensional projection back into the full weight space — needed to reach a target level of task performance when adapting a pretrained model. Introduced to this vault via claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension and extended to the weight-update (rather than the weights) as "intrinsic rank" by LoRA (claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two). Falls as models get larger and better pretrained, a trend Aghajanyan et al. attribute to pre-training acting as an implicit compressor (claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension).

Deliberately kept distinct in this vault from activation-space representation dimensionality, a related but different geometric quantity where safety-relevant concepts live as linear directions — see claim-teo-2025-linear-safety-structure-grows-with-model-size and entity-linear-representation-hypothesis for the boundary. Also one of three domains (alongside monkey motor cortex and an artificial recurrent network) where the vault tracks a recurring "adaptation confined to a low-dimensional subspace" shape — see observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets.

Two constructs, one name (2026-07-25 sharpening). "Intrinsic dimension" denotes at least two different mathematical objects even within machine learning, and this hub is anchored on the weight-space one:

• Weight-space / objective-landscape ID — the codimension of a solution set in parameter space (claim-li-2018-intrinsic-dimension-objective-landscape-codimension-parameter-space, the 2018 origin; applied to fine-tuning by Aghajanyan; operationalized as low-rank weight updates by LoRA). This is what "fine-tuning is low-dimensional" means.
• Representation-space ID — the minimal parametrization of the manifold traced by layer activations (claim-ansuini-2019-two-intrinsic-dimensions-representation-vs-weight-space). This is a property of activity, and it is the same type of object as the neuroscience neural-manifold intrinsic dimension (claim-jazayeri-ostojic-2021-neural-manifold-intrinsic-dimension-parametrizes-activity) — the weight-space notion above is not.

The only located formal bridge between the two ML notions is GeLoRA, which derives representation ID as a lower bound on optimal LoRA rank (claim-gelora-2024-representation-intrinsic-dimension-lower-bounds-lora-rank) — a directed relationship, not an identity, and one that stays entirely on the ML side. Whether the cross-domain recurrence is one object or three analogies remains open at question-low-dimensional-subspace-one-object-or-analogy.

References

• claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension · claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension · claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two · claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size
• Weight-space origin & the two-construct split: claim-li-2018-intrinsic-dimension-objective-landscape-codimension-parameter-space · claim-ansuini-2019-two-intrinsic-dimensions-representation-vs-weight-space · claim-jazayeri-ostojic-2021-neural-manifold-intrinsic-dimension-parametrizes-activity · claim-gelora-2024-representation-intrinsic-dimension-lower-bounds-lora-rank
• observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets · question-low-dimensional-subspace-one-object-or-analogy · question-intrinsic-dimension-falls-with-model-scale-adaptation