The 'intrinsic dimension' behind LLM fine-tuning originates as the codimension of a solution set in weight space, measured by random-subspace training (Li et al. 2018)

The intrinsic-dimension method that Aghajanyan et al. (2020) apply to language-model fine-tuning does not originate in NLP. It comes from Li, Farkhoor, Liu & Yosinski, "Measuring the Intrinsic Dimension of Objective Landscapes" (arXiv:1804.08838, ICLR 2018), whose construction lives entirely in weight space. Given a parameter vector θ ∈ ℝᴰ, they train inside a randomly chosen d-dimensional subspace of the full D-dimensional parameter space rather than the full space itself: "By performing experiments with gradually larger values of d, we can find the subspace dimension at which solutions first appear, which we call the measured intrinsic dimension of a particular problem."

The formal definition is set-geometric, not statistical: "we define the intrinsic dimensionality d_int of a solution as the codimension of the solution set inside of ℝᴰ." The measured object is the objective landscape — the loss surface over network weights — and the quantity is how few free directions in parameter space suffice to reach a target loss. This is precisely the quantity Aghajanyan et al. later measure for fine-tuning (claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension) and that LoRA operationalizes as low-rank weight updates (claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two, entity-lora).

Critically, this is a different mathematical object from a data/representation manifold. It measures nothing about activity, and it is not the same construct as the representation-space intrinsic dimension of a network's activations (claim-ansuini-2019-two-intrinsic-dimensions-representation-vs-weight-space) or the neuroscience neural-manifold intrinsic dimension (claim-jazayeri-ostojic-2021-neural-manifold-intrinsic-dimension-parametrizes-activity), both of which parametrize an activity manifold. The vault's entity-intrinsic-dimension hub is anchored on this weight-space notion; the boundary between it and the activation-space notion is the crux of observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets and question-low-dimensional-subspace-one-object-or-analogy.