Parameter-Efficient Transfer Learning with Diff Pruning
Demi Guo Harvard University、Alexander M. Rush Cornell University、Yoon Kim MIT CSAIL MIT-IBM Watson AI
Abstract
The large size of pretrained networks makes them difficult to deploy for multiple tasks in storage-constrained settings. Diff pruning enables parameter-efficient transfer learning that scales well with new tasks. The approach learns a task-specific “diff” vector that extends the original pretrained parameters. This diff vector is adaptively pruned during training with a differentiable approximation to the $L_{0}$ -norm penalty to encourage sparsity. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task. Since it does not require access to all tasks during training, it is attractive in on-device deployment settings where tasks arrive in stream or even from different providers. Diff pruning can match the performance of finetuned baselines on the GLUE benchmark while only modifying 0.5 $\%$ of the pretrained model’s parameters per task and scales favorably in comparison to popular pruning approaches. †† Code: https://github.com/dguo98/DiffPruning
中文速览
大规模预训练语言模型（如BERT）在迁移到多个下游任务时面临存储瓶颈——每个任务都要保存一套完整的模型参数，代价极高。Diff Pruning 的思路是：保持预训练参数不动，只为每个任务学习一个稀疏的"差分向量"（diff vector）叠加在原参数上，通过可微分的 L₀ 范数近似惩罚在训练中自动压缩这个向量，使绝大多数位置归零。在 GLUE 基准测试上，该方法仅修改预训练模型 0.5% 的参数，就能达到与完整微调相当的性能，远优于 Adapters（约需 3.6% 额外参数）等主流方案。由于每个任务只需独立训练并存储自己的稀疏差分向量，任务之间互不干扰、无需同时可见，这对边缘设备上的流式多任务部署场景具有重要的实用价值。
原文 arXiv:2012.07463;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.07463v2