---
title: "Fine-tuning a large language model has very low intrinsic dimension: 200 parameters tune RoBERTa to 90% of full performance on MRPC (Aghajanyan et al. 2020)"
type: "claim"
status: "seedling"
writer_model: "claude-opus-4-8"
source_url: "https://arxiv.org/abs/2012.13255"
source_title: "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning"
source_author: "Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta"
source_date: "2020-12-22T00:00:00.000Z"
source_quote: "by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC"
source_tier: 1
audit_status: "verified-verbatim (arXiv abstract read directly via WebFetch 2026-07-12 — title 'Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning', authors Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta, submission date 22 Dec 2020, and both the 200-parameter/RoBERTa/90%-MRPC sentence and the 'larger models tend to have lower intrinsic dimension' sentence confirmed verbatim against arxiv.org/abs/2012.13255)."
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-low-dim-subspace-brains-and-nets.md, 2026-07-12 (headless)"
origin: "hop-batch"
derived_from: "10-inbox/raw/2026-07-11-hop-low-dim-subspace-brains-and-nets.md"
date_created: "2026-07-12T00:00:00.000Z"
tags: ["dimensionality","fine-tuning","LoRA","intrinsic-dimension","large-language-models","parameter-efficient"]
---


Aghajanyan, Zettlemoyer & Gupta, "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" (arXiv:2012.13255, 2020), measure the *intrinsic dimension* of adapting a pretrained language model — the smallest number of free parameters, in a random low-dimensional projection back into the full weight space, needed to reach a target performance. It is startlingly small: "by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC." Fine-tuning a model with hundreds of millions of weights is, effectively, a low-dimensional operation.

The paper also reports a scaling regularity: "larger models tend to have lower intrinsic dimension after a fixed number of pre-training updates, at least in part explaining their extreme effectiveness" — i.e. bigger, better-pretrained models are *geometrically easier* to adapt, not harder. That counter-intuitive scale relationship was pursued at [[question-intrinsic-dimension-falls-with-model-scale-adaptation]], now answered: the mechanism is pre-training acting as an implicit compressor ([[claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension]]), the trend holds out to GPT-3 175B ([[claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two]]), and the "geometrically safer" inference it invites does *not* hold — see [[claim-teo-2025-linear-safety-structure-grows-with-model-size]] and [[claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size]].

> Bridge (2026-07-18): the falling-intrinsic-dimension trend and its safety implications are chased in full at the four notes linked above.

This intrinsic-dimension result is the geometry that parameter-efficient methods such as [[entity-lora|LoRA]] later operationalize by restricting weight updates to a low-rank subspace. It is also the large-language-model leg of a cross-domain recurrence: the same "adaptation lives in a low-dimensional subspace" shape appears in monkey motor cortex ([[claim-sadtler-2014-within-manifold-bci-learning-fast-outside-resists]]) and in an artificial recurrent network ([[claim-feulner-clopath-2021-rnn-reproduces-manifold-learning-asymmetry]]). Whether these are one mathematical object or three analogies is treated in [[observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets]]; the originating LLM-introspection thread is [[claim-llm-neural-metacognition-incomplete]].