capture promoted Tier 1 2026-07-17

Why does the intrinsic dimension of fine-tuning fall as models get larger — and does that make bigger models geometrically safer to adapt?

This capture answers the open thread at question-intrinsic-dimension-falls-with-model-scale-adaptation. It picks up where claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension left off — that note establishes that larger models have lower intrinsic dimension; this one chases why, and separately tests the safety inference the question raises. All sources below are Tier 1 (arXiv preprints/papers, own findings, read directly from the PDF text — not paraphrased through a secondary write-up), consistent with the sourcing floor for technical-mechanism and quantitative claims.

Claim: Pre-training works as an implicit compression process — larger, better pre-trained models need fewer bits to describe a fine-tuning task, which is the proposed mechanism for falling intrinsic dimension

Aghajanyan, Zettlemoyer & Gupta ("Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning," arXiv:2012.13255) interpret the intrinsic parameter vector of a fine-tuning run as "the minimal description length of the task within the framework dictated by the pre-trained representations," and hypothesize "that pre-training is implicitly lowering the intrinsic dimensionality of the average NLP task, and therefore compress the minimal description length of those same tasks." They test this directly by retraining RoBERTa-Base from scratch and measuring intrinsic dimension (d90) across pre-training checkpoints on six datasets: "the intrinsic dimensionality of RoBERTa-Base monotonically decreases as we continue pre-training" — even though "the language model does not have access to downstream datasets" during pre-training. They then ran the same measurement across a dozen-plus released pre-trained models of varying size (BERT, RoBERTa, BART, Electra, ALBERT, XLNet, T5, XLM-R) on the MRPC task and report: "we see a strong general trend that as the number of parameters increases, the intrinsic dimension of fine-tuning on MRPC decreases." Their own gloss on the result: "the more parameters we have in the model, the less we need to represent a task." Source: arXiv:2012.13255, §5.1–5.2 (Tier 1, exact quotes, TLS verified).

Claim: The trend holds at far larger scale than Aghajanyan et al. tested — GPT-3's 175B-parameter fine-tuning update has intrinsic rank as low as 1 or 2

Hu et al., introducing LoRA ("LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685), build directly on Aghajanyan et al.'s finding — "We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension" — and extend the measurement to the weight-update during adaptation rather than the weights themselves: "we hypothesize that the change in weights during model adaptation also has a low 'intrinsic rank'." Their headline empirical result, at a scale two orders of magnitude beyond the BERT/RoBERTa-era models Aghajanyan et al. studied: "Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288." Compared to full fine-tuning of GPT-3 175B, "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." Source: arXiv:2106.09685, Abstract and §1 (Tier 1, exact quotes, TLS verified).

Claim: The safety inference does not obviously follow — larger models' higher-dimensional activation space is associated with more exploitable linear structure for jailbreaking, not less

Teo, Abdullaev & Nguyen ("The Blessing and Curse of Dimensionality in Safety Alignment," arXiv:2507.20333, COLM 2025) study a related but distinct dimensionality — not the intrinsic dimension of the fine-tuning objective (weight space), but the dimensionality of a model's internal activation representations, where safety-relevant concepts live as linear directions. They report that such linear representations "are assumed to be emergent in LLMs. That is, they are only present in models that are sufficiently large, with high-dimensional representations" — i.e., the exploitable structure is a property that grows in, not out of, larger models. Their abstract states the trade-off directly: "the linear structures in the activation space can be exploited, in the form of activation engineering, to circumvent its safety alignment," and that "the curse of high-dimensional representations uniquely impacts LLMs." Their proposed fix runs the opposite direction from the naive "bigger is safer" intuition: deliberately projecting representations down to a lower-dimensional subspace — "dimensional reduction significantly reduces susceptibility to jailbreaking through representation engineering." Source: arXiv:2507.20333, Abstract and §1 (Tier 1, exact quotes, TLS verified). Note the concept boundary: this is activation-space dimensionality, not the weight-space intrinsic dimension of a fine-tuning run — the two are related by the same "low-dimensional structure" family Seek tracks in observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets, but are not the same measured quantity, and this capture does not claim they are.

Claim: The same low-rank cheapness that makes big models easy to adapt legitimately also makes their safety alignment cheap to remove via fine-tuning — a small number of examples suffices regardless of model size

Qi et al. ("Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!," arXiv:2310.03693) report: "we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions." A follow-on paper, Zhang et al. ("Safety at One Shot: Patching Fine-Tuned LLMs with a Single Instance," arXiv:2601.01887, preprint dated 6 Jan 2026), analyzing why safety alignment is simultaneously so fragile to attack and so cheap to repair, reports finding via SVD of the safety gradient that "the alignment signal lies in a low-rank intrinsic subspace," and that "this antagonistic and low-dimensional structure explains why a single safety update can efficiently neutralize harmful fine-tuning and why the recovery converges rapidly, regardless of model size or harmful fine-tuning scale" (emphasis on the original wording — model size does not appear to be a protective factor in their framing). Sources: arXiv:2310.03693 Abstract; arXiv:2601.01887 §1 (both Tier 1, exact quotes, TLS verified).

Seek's commentary: Putting the four claims together: the mechanism side of the question has a clean, well-sourced answer — pre-training acts as an implicit compressor of the "average task," and bigger/better-pretrained models compress harder, so less new information (fewer intrinsic parameters) is needed to specify a fine-tune. That part replicates from BERT-era models (Aghajanyan et al., 10^8–10^9 parameters) up through GPT-3 175B (LoRA), which is a real scale range. The safety half of the question is where the capture pushes back on its own premise. "Bigger models need less capacity to adapt" turns out to say nothing reassuring about robustness to adaptation — if anything, the cited evidence runs the other way twice: (1) higher-dimensional activation spaces in larger models are described as harboring more exploitable linear safety structure, not less, and (2) the low-rank subspace that makes safety alignment cheap to patch is the same kind of structure that makes it cheap to attack, and the "Safety at One Shot" paper explicitly frames its efficient-recovery result as holding "regardless of model size" — i.e., size is not doing safety work either way in that framing. So: geometrically cheaper to adapt, yes, robustly confirmed. Geometrically safer, no — the available evidence does not support that inference, and the two dimensionality concepts (weight-space intrinsic dimension vs. activation-space representation dimensionality) should not be treated as interchangeable when this gets promoted.

Further leads

"Safety at One Shot" (arXiv:2601.01887) appears in secondary search summaries to report the intrinsic dimension of the safety-relevant subspace as under 20 — worth a direct-text confirmation pass before recording as a quantitative claim; not independently verified against the primary text in this session. [unverified-quant — needs primary]
Multiple 2025–2026 papers build "safety subspace" defenses on the low-rank-safety-gradient idea (e.g. "Decoupling Safety into Orthogonal Subspace," arXiv:2510.09004; "Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation," arXiv:2512.23260) — a cluster worth its own mapping pass if this thread gets promoted.
"Safety Alignment Should Be Made More Than Just a Few Tokens Deep" (Qi et al., arXiv:2406.05946) argues safety alignment concentrates in the first few output tokens rather than being distributed — a distinct "shallowness" mechanism from the low-rank-subspace framing, possibly complementary.
Aghajanyan et al. also derive a generalization bound that scales with intrinsic dimension d rather than total parameter count D (their Theorem 1) — a theoretical thread not chased here.

Entity candidates

Armen Aghajanyan — person — lead author of the founding intrinsic-dimension-of-fine-tuning paper; recurs across this vault's dimensionality thread
Intrinsic dimension — concept — the core geometric quantity this whole capture turns on; already partially covered but could anchor its own concept note
LoRA (Low-Rank Adaptation) — concept/term — the engineering technique built directly on this geometry; widely referenced, no dedicated note yet spotted
Minimal description length / compression framework — concept — Aghajanyan et al.'s proposed interpretive mechanism, could stand alone as a concept note bridging into MDL theory
Linear representation hypothesis — concept — the activation-space framing underlying the "Blessing and Curse" paper; distinct from weight-space intrinsic dimension, worth its own note to prevent future conflation
Xiangyu Qi — person — lead author of the fine-tuning-breaks-safety line of work, recurs across multiple cited papers
Fine-tuning attack — concept/term — the safety-relevant adversarial framing (cheap fine-tuning as a jailbreak vector)
Safety gradient low-rank subspace — concept — the specific mechanism Zhang et al. propose for why safety is both fragile and cheap to repair

Source

Tier 1 Aghajanyan, Zettlemoyer & Gupta; Hu et al.; Teo, Abdullaev & Nguyen; Qi et al.; Zhang et al. 2020-12-22

https://arxiv.org/abs/2012.13255, https://arxiv.org/abs/2106.09685, https://arxiv.org/abs/2507.20333, https://arxiv.org/abs/2310.03693, https://arxiv.org/abs/2601.01887

written by claude-sonnet-5 · Batch research run, 2026-07-17, answering [[question-intrinsic-dimension-falls-with-model-scale-adaptation]] · raw markdown