---
title: "LoRA shows GPT-3 175B's fine-tuning weight update has intrinsic rank as low as 1 or 2, extending the falling-intrinsic-dimension trend two orders of magnitude past Aghajanyan et al.'s test range"
type: "claim"
status: "seedling"
writer_model: "claude-sonnet-5"
source_url: "https://arxiv.org/abs/2106.09685"
source_title: "LoRA: Low-Rank Adaptation of Large Language Models"
source_author: "Hu et al."
source_date: "2021-06"
source_quote: "Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient."
source_tier: 1
audit_status: "capture-verified (bee read arXiv:2106.09685 Abstract and §1 directly at capture time, exact quotes, TLS verified; not independently re-fetched this promotion pass, headless). 2026-07-19 cross-model audit (claude-fable-5): independently re-fetched arXiv:2106.09685 — abstract via arxiv.org, §1 via extract_pdf of v2 PDF (sha256 e9a0d312…, TLS verified); all quoted material confirmed verbatim in Abstract/§1; two corrections applied: (1) restored the truncated end of the §1 quote (', making LoRA both storage- and compute-efficient' — the capture had substituted a period at '12,288'), (2) fixed the abstract-baseline paraphrase 'full fine-tuning of GPT-3 175B' to the paper's 'GPT-3 175B fine-tuned with Adam', which the 3x GPU-memory figure depends on. Tier 1 confirmed; all four wikilink targets resolve."
provenance: "Promotion from 10-inbox/raw/2026-07-17-why-does-the-intrinsic-dimension-of-fine-tuning.md, 2026-07-18 (headless)"
origin: "batch"
derived_from: "10-inbox/raw/2026-07-17-why-does-the-intrinsic-dimension-of-fine-tuning.md"
date_created: "2026-07-18T00:00:00.000Z"
tags: ["dimensionality","fine-tuning","intrinsic-dimension","LoRA","scaling","large-language-models","parameter-efficient","GPT-3"]
audits: ["2026-07-19 claude-fable-5"]
---


Hu et al., introducing LoRA ("LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685), build directly on [[claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension]] — "We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low [[entity-intrinsic-dimension|intrinsic dimension]]" — but shift the measurement from the intrinsic dimension of the *weights* to the intrinsic *rank* of the weight *update* during adaptation: "we hypothesize that the change in weights during model adaptation also has a low 'intrinsic rank'."

Their headline result, at a scale two orders of magnitude beyond the BERT/RoBERTa-era models (10⁸–10⁹ parameters) Aghajanyan et al. tested: "Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient." Compared to "GPT-3 175B fine-tuned with Adam" (the abstract's baseline — the optimizer matters, since Adam's optimizer states are much of the memory), "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times."

The result matters as a scale check on Aghajanyan et al.'s trend, not merely an engineering technique: the falling-intrinsic-dimension pattern that motivated [[question-intrinsic-dimension-falls-with-model-scale-adaptation]] holds all the way from BERT-scale models to GPT-3 175B, the largest model in this vault's current dimensionality thread. See [[entity-lora]] and [[entity-armen-aghajanyan]].

> [!note] Seek's commentary:
> Rank one or two, for a matrix whose full rank is 12,288 — that's not "lower-dimensional," that's barely a dimension at all, and it's the number that makes the Aghajanyan trend feel less like a regularity between similarly-sized transformer variants and more like a genuine law of scale. It's also the number the next two notes in this cluster spend their whole argument complicating: cheap-to-adapt and safe-to-adapt turn out to be different claims wearing the same low-rank clothing. — Seek