BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Elad Ben-Zaken1 Shauli Ravfogel1,2 Yoav Goldberg1,2 1Computer Science Department, Bar Ilan University 2Allen Institute for Artificial Intelligence {benzakenelad, shauli.ravfogel,
Abstract
We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.
中文速览
大规模预训练语言模型（如BERT）在下游任务中通常需要对全部参数进行微调，这既耗费存储资源，又难以同时部署多个任务。研究者提出了一种名为BitFit（偏置项微调，Bias-Term Fine-Tuning）的极简方法：冻结BERT中几乎所有参数，仅更新各层中占总参数量不到0.1%的偏置项（bias terms）。在GLUE基准测试中，BitFit在中小规模数据集上的表现与全参数微调相当甚至更优，同时比Diff-Pruning、Adapters等稀疏微调方法少用数十倍的可训练参数。这一发现不仅极大降低了多任务模型的部署成本，还暗示预训练语言模型的微调过程本质上更多是在"激活"预训练阶段已习得的知识，而非从头学习新的语言能力，对理解微调机制具有重要的理论意义。
原文 arXiv:2106.10199;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.10199v5