Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Mitchell A. Gordon、Kevin Duh、Nicholas Andrews Johns Hopkins University
Abstract
Pre-trained feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. While effective, these feature extractors may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.
中文速览
大规模预训练语言模型BERT在众多自然语言处理任务中表现出色，但动辄数亿参数的体量让它在手机等资源受限的设备上难以部署。研究者对BERT进行了权重幅度剪枝（magnitude weight pruning）实验，系统考察压缩程度如何影响模型向下游任务的迁移能力。结果发现三个明显的"剪枝区间"：剪掉30–40%的权重几乎不损失任何性能；剪枝比例进一步提高则会破坏预训练阶段积累的语言知识，令下游任务的表现随预训练损失的上升而线性下滑；剪枝比例极高时（约70%以上），模型连在下游数据集上重新拟合的能力也会受损。此外，先在特定任务上微调再剪枝，并不比直接在预训练阶段剪枝更有优势。这意味着只需在预训练时统一压缩一次BERT，就能在不重复实验的前提下将轻量版模型分发给所有下游任务，大幅降低计算成本与碳排放。
原文 arXiv:2002.08307;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.08307v2