Fine-tuning a large language model has very low intrinsic dimension: 200 parameters tune RoBERTa to 90% of full performance on MRPC (Aghajanyan et al. 2020)

Aghajanyan, Zettlemoyer & Gupta, "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" (arXiv:2012.13255, 2020), measure the intrinsic dimension of adapting a pretrained language model — the smallest number of free parameters, in a random low-dimensional projection back into the full weight space, needed to reach a target performance. It is startlingly small: "by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC." Fine-tuning a model with hundreds of millions of weights is, effectively, a low-dimensional operation.

The paper also reports a scaling regularity: "larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness" — i.e. bigger, better-pretrained models are geometrically easier to adapt, not harder. That counter-intuitive scale relationship was pursued at question-intrinsic-dimension-falls-with-model-scale-adaptation, now answered: the mechanism is pre-training acting as an implicit compressor (claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension), the trend holds out to GPT-3 175B (claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two), and the "geometrically safer" inference it invites does not hold — see claim-teo-2025-linear-safety-structure-grows-with-model-size and claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size.

Bridge (2026-07-18): the falling-intrinsic-dimension trend and its safety implications are chased in full at the four notes linked above.

This intrinsic-dimension result is the geometry that parameter-efficient methods such as LoRA later operationalize by restricting weight updates to a low-rank subspace. It is also the large-language-model leg of a cross-domain recurrence: the same "adaptation lives in a low-dimensional subspace" shape appears in monkey motor cortex (claim-sadtler-2014-within-manifold-bci-learning-fast-outside-resists) and in an artificial recurrent network (claim-feulner-clopath-2021-rnn-reproduces-manifold-learning-asymmetry). Whether these are one mathematical object or three analogies is treated in observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets; the originating LLM-introspection thread is claim-llm-neural-metacognition-incomplete.