Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta Facebook
Abstract
Although pretrained language models can be fine-tuned to produce state-of-the-art results for a very wide range of language understanding tasks, the dynamics of this process are not well understood, especially in the low data regime. Why can we use relatively vanilla gradient descent algorithms (e.g., without strong regularization) to tune a model with hundreds of millions of parameters on datasets with only hundreds or thousands of labeled examples? In this paper, we argue that analyzing fine-tuning through the lens of intrinsic dimension provides us with empirical and theoretical intuitions to explain this remarkable phenomenon. We empirically show that common pre-trained models have a very low intrinsic dimension; in other words, there exists a low dimension reparameterization that is as effective for fine-tuning as the full parameter space. For example, by optimizing only 200 trainable parameters randomly projected back into the full space, we can tune a RoBERTa model to achieve 90% of the full parameter performance levels on MRPC. Furthermore, we empirically show that pre-training implicitly minimizes intrinsic dimension and, perhaps surprisingly, larger models tend to have lo
中文速览
预训练语言模型（如BERT、RoBERTa）在小数据集上微调效果出奇地好，但背后原因一直不清楚。这篇论文借助"内在维度"（intrinsic dimension）这把钥匙来解释这一现象——内在维度衡量的是：把一个优化问题压缩到多低维的子空间里，仍然能解得足够好。实验结果令人震惊：只需在随机投影回全参数空间的区区200个参数上做优化，RoBERTa模型在MRPC任务上就能达到全参数微调90%的性能；更大的预训练模型反而拥有更低的内在维度，说明模型规模越大、预训练越充分，下游任务就越容易被"压缩"表达。论文还把内在维度与压缩式泛化界（compression-based generalization bound）相连接，给出了与全参数量无关的泛化保证，从理论上解释了为什么这类方法在小样本场景下依然能泛化得如此之好。
原文 arXiv:2012.13255;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.13255v1