Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Mengjie Zhao†*, Tao Lin‡*, Fei Mi‡, Martin Jaggi‡, Hinrich Schütze† † LMU Munich, Germany ‡ EPFL, Switzerland {tao.lin, fei.mi, * Equal contribution.
Abstract
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT, RoBERTa, and DistilBERT on eleven diverse NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred. Intrinsic evaluations show that representations computed by our binary masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.
中文速览
预训练语言模型（如BERT）的标准微调（finetuning）需要为每个任务单独保存数亿个浮点参数，多任务部署时内存开销极大。研究者提出了一种"掩码（masking）"方案：完全不改动预训练权重，而是为每个下游任务学习一套二值掩码（binary mask），通过端到端训练决定保留或丢弃哪些预训练参数。在BERT、RoBERTa和DistilBERT上跨11项NLP任务的实验表明，该方法性能与微调相当，但每个任务只需存储1比特的掩码而非32位浮点数，内存占用仅约为微调的3%。进一步分析损失曲面和内部表征后发现，掩码与微调所得模型落在几乎可以用一条测试准确率近似恒定的线段相连的极小值处，说明预训练权重已蕴含足够的任务信息，掩码是一种切实可行且高度参数高效的微调替代方案。
原文 arXiv:2004.12406;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.12406v2