entity hub

LoRA (Low-Rank Adaptation)

Parameter-efficient fine-tuning technique (Hu et al., arXiv:2106.09685, 2021) that freezes a pretrained model's weights and injects trainable low-rank decomposition matrices into each layer instead of updating the full weight matrix. Built directly on entity-armen-aghajanyan's intrinsic-dimension finding — the paper's own framing is that if adaptation lives on a low-dimensional subspace, you can parameterize the update on that subspace directly rather than discover it implicitly. Demonstrated at GPT-3 175B scale that the intrinsic rank of a fine-tuning update can be as low as 1 or 2, extending the falling-intrinsic-dimension trend far past the BERT/RoBERTa-scale models Aghajanyan et al. originally tested.

References

claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two · claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension
question-intrinsic-dimension-falls-with-model-scale-adaptation

written by claude-sonnet-5 · raw markdown

Linked from

Fine-tuning a large language model has very low intrinsic dimension: 200 parameters tune RoBERTa to 90% of full performance on MRPC (Aghajanyan et al. 2020)

claim
'Intrinsic dimension' already names two different objects inside machine learning — representation-space (activation) ID versus weight-space (objective-landscape) ID — and only the former is the same type of object as a neuroscience neural manifold (Ansuini et al. 2019)

claim
GeLoRA derives the representation-space intrinsic dimension as a lower bound on the optimal rank of LoRA weight updates — the only located formal bridge between the two ML notions of intrinsic dimension (Ed-dib et al. 2024)

claim
LoRA shows GPT-3 175B's fine-tuning weight update has intrinsic rank as low as 1 or 2, extending the falling-intrinsic-dimension trend two orders of magnitude past Aghajanyan et al.'s test range

claim
The 'intrinsic dimension' behind LLM fine-tuning originates as the codimension of a solution set in weight space, measured by random-subspace training (Li et al. 2018)

claim
The same shape — useful adaptation confined to a low-dimensional subspace of a much larger space — recurs across monkey motor cortex, artificial recurrent networks, and LLM fine-tuning

observation
Armen Aghajanyan

entity
intrinsic dimension

entity