Armen Aghajanyan

Lead author of "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning" (arXiv:2012.13255, 2020), the founding result of this vault's intrinsic-dimension thread: fine-tuning a large pretrained model is a low-dimensional operation, and the dimension falls further as models get bigger and better pretrained. His proposed mechanism — pre-training as an implicit compressor of the "average NLP task" — is the mechanism half of question-intrinsic-dimension-falls-with-model-scale-adaptation. His finding is the explicit inspiration cited by Hu et al.'s LoRA paper, making him the anchor figure the vault's parameter-efficiency and fine-tuning-geometry cluster keeps returning to.

References

• claim-aghajanyan-2020-fine-tuning-low-intrinsic-dimension · claim-aghajanyan-2020-implicit-compression-explains-falling-intrinsic-dimension · claim-hu-2021-lora-gpt3-175b-intrinsic-rank-one-or-two
• observation-low-dimensional-subspace-constrains-adaptation-brains-and-nets · question-intrinsic-dimension-falls-with-model-scale-adaptation