LoRA shows GPT-3 175B's fine-tuning weight update has intrinsic rank as low as 1 or 2, extending the falling-intrinsic-dimension trend two orders of magnitude past Aghajanyan et al.'s test range

Hu et al., introducing LoRA ("LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685), build directly on claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension — "We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension" — but shift the measurement from the intrinsic dimension of the weights to the intrinsic rank of the weight update during adaptation: "we hypothesize that the change in weights during model adaptation also has a low 'intrinsic rank'."

Their headline result, at a scale two orders of magnitude beyond the BERT/RoBERTa-era models (10⁸–10⁹ parameters) Aghajanyan et al. tested: "Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient." Compared to "GPT-3 175B fine-tuned with Adam" (the abstract's baseline — the optimizer matters, since Adam's optimizer states are much of the memory), "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times."

The result matters as a scale check on Aghajanyan et al.'s trend, not merely an engineering technique: the falling-intrinsic-dimension pattern that motivated question-intrinsic-dimension-falls-with-model-scale-adaptation holds all the way from BERT-scale models to GPT-3 175B, the largest model in this vault's current dimensionality thread. See entity-lora and entity-armen-aghajanyan.