Movement Pruning: Adaptive Sparsity by Fine-Tuning
Victor Sanh1, Thomas Wolf1, Alexander M. Rush1,2 1Hugging Face, 2Cornell University ;
Abstract
Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.
中文速览
大规模预训练语言模型（如BERT）在迁移学习场景下用传统幅度剪枝（magnitude pruning）效果很差，因为那些方法只看权重的绝对值大小，而BERT的权重在微调前后变化极小，几乎在预训练阶段就已决定了哪些权重会被剪掉，根本没有利用到下游任务的信息。为此，作者提出"移动剪枝"（movement pruning）——不看权重本身的大小，而是追踪微调过程中权重"向零靠近还是远离零"的动态变化，把那些持续收缩趋零的权重剪掉，同时保留正在远离零的权重，从而让剪枝决策真正由下游任务数据驱动。实验在SQuAD问答和MNLI自然语言推理等任务上表明，在高稀疏度（保留权重不足15%）条件下，移动剪枝显著优于幅度剪枝和L₀正则化等方法；若进一步结合知识蒸馏，仅保留3%的模型参数就能把精度损失控制在极小范围内。这项工作为在边缘设备上高效部署大模型提供了一条切实可行的压缩路径，并从理论和实验两方面说明了"一阶"剪枝信号在迁移学习场景下的根本优势。
原文 arXiv:2005.07683;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.07683v2