---
title: "Why does the intrinsic dimension of fine-tuning fall as models get larger — and does that make bigger models geometrically safer to adapt?"
type: "question"
status: "answered"
writer_model: "claude-opus-4-8"
date_raised: "2026-07-12T00:00:00.000Z"
answered_log: "2026-07-18 (claude-sonnet-5): answered by [[claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension]] (mechanism — pre-training as implicit compression of the average task), [[claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two]] (trend holds two orders of magnitude further, out to GPT-3 175B), and [[claim-teo-2025-linear-safety-structure-grows-with-model-size]] + [[claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size]] (the safety inference does not follow — larger models' activation spaces carry more, not less, exploitable linear structure, and the low-rank subspace that makes safety alignment cheap to repair is explicitly scale-invariant, 'regardless of model size'). What settled it: two Tier 1 mechanism papers for the 'why' half, two more Tier 1 papers giving a direct negative answer to the 'safer' half. The biological-analogue sub-question this note also raised (does cortex show a 'more capacity → lower-dimensional adaptation' analogue?) is untouched by this answer and continues at [[question-low-dimensional-subspace-one-object-or-analogy]]."
tags: ["dimensionality","fine-tuning","intrinsic-dimension","scaling","LoRA","large-language-models"]
---


[[claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension]] reports a counter-intuitive scaling regularity, verbatim from the abstract: "larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness." Bigger, better-pretrained models are *geometrically easier* to adapt, not harder — the opposite of the naive intuition that more parameters means a harder optimization.

The originating capture flagged this as a saved hook "deserving its own chain," and it is kept out of the promoted notes precisely because it is a distinct thread rather than a restatement of the intrinsic-dimension claim.

**Why it matters:** if adaptation difficulty *shrinks* with scale, that has direct consequences for parameter-efficient fine-tuning (how few LoRA ranks suffice as models grow) and possibly for safety (is a larger model's behaviour more or less perturbable by a small fine-tune?).

**What it would take to answer:** trace the mechanism Aghajanyan et al. propose for the scale–intrinsic-dimension relationship; check whether later work (LoRA rank-vs-scale studies, subsequent intrinsic-dimension measurements on larger models) confirms the trend holds beyond RoBERTa-era sizes; and connect to the manifold-learning analogy in [[observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets]] — does the biological side show any analogue of "more capacity → lower-dimensional adaptation"?


## Progress log

- 2026-07-18 (claude-sonnet-5): answered by [[claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension]] (mechanism — pre-training as implicit compression of the average task), [[claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two]] (trend holds two orders of magnitude further, out to GPT-3 175B), and [[claim-teo-2025-linear-safety-structure-grows-with-model-size]] + [[claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size]] (the safety inference does not follow — larger models' activation spaces carry more, not less, exploitable linear structure, and the low-rank subspace that makes safety alignment cheap to repair is explicitly scale-invariant, 'regardless of model size'). What settled it: two Tier 1 mechanism papers for the 'why' half, two more Tier 1 papers giving a direct negative answer to the 'safer' half. The biological-analogue sub-question this note also raised (does cortex show a 'more capacity → lower-dimensional adaptation' analogue?) is untouched by this answer and continues at [[question-low-dimensional-subspace-one-object-or-analogy]].