TinyBERT: Distilling BERT for Natural Language Understanding
Xiaoqi Jiao1 , Yichun Yin2∗2{}^{2\hskip 0.3pt*}\hskip 0.3pt\thanks{\hskip 0.5pt Corresponding authors.}, Lifeng Shang2‡, Xin Jiang2 Xiao Chen2, Linlin Li3, Fang Wang1‡ and Qun Liu2 1Key Laboratory of Information Storage System, Huazhong University of Science and Technology, Wuhan National Laboratory for Optoelectronics 2Huawei Noah’s Ark Lab 3Huawei Technologies Co., Ltd. Authors contribute equally. This work is done when Xiaoqi Jiao is an intern at Huawei Noah’s Ark Lab. Corresponding authors.
Abstract
Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be effectively transferred to a small “student” TinyBERT. Then, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT.
中文速览
大规模预训练语言模型（如BERT）虽然效果强大，却因参数量庞大而难以部署到手机等资源受限的设备上。为此，研究者提出了TinyBERT，设计了一套专门针对Transformer结构的知识蒸馏（knowledge distillation）方法，让小模型（学生）同时学习大模型（教师BERT）的注意力矩阵、隐层状态、嵌入层输出和预测层输出，从而把教师模型中蕴含的语言知识尽可能完整地传递出去。此外，他们还引入了"两阶段学习框架"：先在通用语料上做预训练蒸馏，再在具体任务上结合数据增强做任务蒸馏，确保学生模型既吸收通用语言知识，又掌握任务专属知识。实验结果显示，仅有4层的TinyBERT在GLUE基准上达到了教师BERT BASE 96.8%以上的性能，体积却缩小7.5倍、推理速度加快9.4倍，6层版本更能与教师模型持平。这项工作为在边缘设备上高效运行强大语言模型提供了切实可行的解决方案，对NLP模型轻量化部署具有重要的实用价值。
原文 arXiv:1909.10351;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.10351v5