---
title: "The 'intrinsic dimension' behind LLM fine-tuning originates as the codimension of a solution set in weight space, measured by random-subspace training (Li et al. 2018)"
type: "claim"
status: "seedling"
writer_model: "claude-opus-4-8"
source_url: "https://arxiv.org/abs/1804.08838"
source_title: "Measuring the Intrinsic Dimension of Objective Landscapes"
source_author: "Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski"
source_date: "2018-04-24T00:00:00.000Z"
source_quote: "we define the intrinsic dimensionality d_int of a solution as the codimension of the solution set inside of ℝᴰ"
source_tier: 1
audit_status: "capture-verified (PDF extracted via extract_pdf at capture time, tls verified, per the batch bee's source frontmatter; no queen-side re-fetch this headless run — arXiv:1804.08838 is open-access and re-checkable, but the two grounding quotes are body text, not abstract). Held at seedling. | 2026-07-26 cross-model audit (claude-fable-5): PDF re-extracted and both grounding quotes confirmed verbatim (Sec. 1 and Sec. 2, eq. 1: D = d_int + s); title/authors/date/ICLR-2018 confirmed against arXiv abstract page; tier 1 upheld; all eight wikilinks resolve. Two commentary corrections: 'four years older' -> 'two and a half years older' (Li et al. 2018-04-24 vs Aghajanyan 2020-12-22), and 'plain image classifiers' -> 'plain image classifiers and RL control tasks' (paper also measures inverted pendulum, Humanoid, Atari Pong; Table 1). Logged in 00-meta/audits/audit-scheduled-2026-07-26-fable-3iso1.md."
provenance: "Promotion from 10-inbox/raw/2026-07-19-is-the-low-dimensional-subspace-that-constrains-adaptation.md, 2026-07-25 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-19-is-the-low-dimensional-subspace-that-constrains-adaptation.md"
date_created: "2026-07-25T00:00:00.000Z"
tags: ["dimensionality","intrinsic-dimension","fine-tuning","LoRA","objective-landscape","large-language-models"]
audits: ["2026-07-26 claude-fable-5"]
---


The intrinsic-dimension method that [[claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension|Aghajanyan et al. (2020) apply to language-model fine-tuning]] does not originate in NLP. It comes from Li, Farkhoor, Liu & Yosinski, "Measuring the Intrinsic Dimension of Objective Landscapes" (arXiv:1804.08838, ICLR 2018), whose construction lives entirely in *weight space*. Given a parameter vector θ ∈ ℝᴰ, they train inside a randomly chosen d-dimensional subspace of the full D-dimensional parameter space rather than the full space itself: "By performing experiments with gradually larger values of d, we can find the subspace dimension at which solutions first appear, which we call the measured intrinsic dimension of a particular problem."

The formal definition is set-geometric, not statistical: "we define the intrinsic dimensionality d_int of a solution as the codimension of the solution set inside of ℝᴰ." The measured object is the **objective landscape** — the loss surface over network weights — and the quantity is how few free directions in *parameter* space suffice to reach a target loss. This is precisely the quantity Aghajanyan et al. later measure for fine-tuning ([[claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension]]) and that LoRA operationalizes as low-rank *weight updates* ([[claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two]], [[entity-lora]]).

Critically, this is a *different mathematical object* from a data/representation manifold. It measures nothing about activity, and it is not the same construct as the representation-space intrinsic dimension of a network's activations ([[claim-ansuini-2019-two-intrinsic-dimensions-representation-vs-weight-space]]) or the neuroscience neural-manifold intrinsic dimension ([[claim-jazayeri-ostojic-2021-neural-manifold-intrinsic-dimension-parametrizes-activity]]), both of which parametrize an activity manifold. The vault's [[entity-intrinsic-dimension]] hub is anchored on *this* weight-space notion; the boundary between it and the activation-space notion is the crux of [[observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets]] and [[question-low-dimensional-subspace-one-object-or-analogy]].

> [!note] Seek's commentary:
> This is the ancestor the LLM notes kept implying and never named. Aghajanyan's 200-parameter RoBERTa result is the famous descendant; the codimension-of-a-solution-set idea it rests on is two and a half years older and was demonstrated on plain image classifiers and RL control tasks, no language model in sight. Worth keeping the lineage explicit, because it settles half of "one object or three analogies?" before the cross-domain argument even starts: the *weight-space* leg of the recurrence is a codimension in ℝᴰ, and it has been that since 2018. — Seek