MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wenhui Wang Hangbo Bao Shaohan Huang Li Dong Furu Wei Microsoft Research Contact person.
Abstract
We generalize deep self-attention distillation in MiniLM Wang et al. (2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student’s attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models111Distilled models and code will be publicly available at https://aka.ms/minilm. distilled from base-size and large-size teach
中文速览
预训练Transformer模型（如BERT、RoBERTa）虽然效果强大，但参数量庞大、推理延迟高，难以在实际场景中部署，因此需要将其压缩成更小的学生模型。本文在MiniLM的基础上提出MiniLMv2，核心创新是用"多头自注意力关系（multi-head self-attention relation）"——即查询、键、值向量之间的缩放点积——作为知识蒸馏的监督信号，取代原先依赖注意力分布的做法，从而彻底解除了学生模型必须与教师模型拥有相同注意力头数的限制，同时也带来更细粒度的自注意力知识。研究还系统考察了教师层的选取策略，发现对大尺寸教师（24层）而言，使用中上层而非最后一层效果更好。实验表明，从BERT、RoBERTa、XLM-R等单语和多语教师模型蒸馏出的学生模型在多项基准上均超越此前最优方法，推理速度最高提升2倍，这为在资源受限场景下高效部署强大语言模型提供了一条更灵活、更通用的压缩路径。
原文 arXiv:2012.15828;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.15828v2