On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
Ruidan He11{}^{~{}1}, Linlin Liu∗12 , Hai Ye∗3, Qingyu Tan13†, Bosheng Ding12†, Liying Cheng14†, Jia-Wei Low12†, Lidong Bing1, Luo Si1 1DAMO Academy, Alibaba Group 2Nanyang Technological University 3National University of Singapore 4Singapore University of Technology and Design ∗ Equally Contributed† Linlin, Qingyu, Bosheng, Liying, and Jia-wei are under the Joint PhD Program between Alibaba and their corresponding universities.
Abstract
Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.
中文速览
适配器微调（Adapter-based tuning）虽已被证明参数效率高，但它究竟在效果上有没有实质优势，此前几乎无人深究。本文系统研究了这一问题：首先用表示相似性分析（Representational Similarity Analysis）证明，适配器微调因冻结预训练模型原有参数，能更好地保留原始知识、缓解"灾难性遗忘"；随后在单语低资源分类、GLUE基准低资源场景以及零样本跨语言迁移等多个任务上，将其与全量微调进行了大规模对比实验。结果表明，适配器微调在低资源和跨语言任务上持续超越全量微调，同时对学习率变化更不敏感、训练更稳定、泛化能力更强；而在高资源场景下两者表现相近，因为此时模型容量的优势比防遗忘的正则化效果更关键。这项工作揭示了适配器微调在效果层面的独特价值，为如何选择预训练语言模型的适配策略提供了重要依据。
原文 arXiv:2106.03164;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.03164v1