MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
Zhiqing Sun1 , Hongkun Yu2, Xiaodan Song2, Renjie Liu2, Yiming Yang1, Denny Zhou2 1Carnegie Mellon University {zhiqings, 2Google Brain {hongkuny, xiaodansong, renjieliu, This work was done when the first author was an intern at Google Brain.
Abstract
Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of $\text{BERT}_{\text{LARGE}}$ , while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated $\text{BERT}_{\text{LARGE}}$ model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3 $\times$ smaller and 5.5 $\times$ faster than $\text{BERT}_{\text{BASE}}$ while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE scor
中文速览
手机等移动端设备算力有限，而 BERT 这类预训练语言模型体积庞大、推理缓慢，根本无法直接部署。为此，研究者提出了 MobileBERT——一个与 BERT 层数相同但每层更"细长"的压缩版模型，通过在每个模块内引入瓶颈结构（bottleneck）并叠加多个前馈网络来保持表达能力，同时先训练一个配套的"倒置瓶颈"教师模型（IB-BERT），再利用逐层特征图对齐和注意力分布对齐的知识迁移方法将其压缩为学生模型。最终，MobileBERT 的体积比 BERT-BASE 小 4.3 倍、速度快 5.5 倍，在 Pixel 4 手机上仅需 62 毫秒完成推理，GLUE 得分仅低 0.6 个点，在 SQuAD 问答任务上甚至还略有超越。这项工作填补了"任务无关的轻量级预训练模型"这一空白，让开发者无需为每个下游任务单独蒸馏，直接在手机端微调即可，具有很强的实用价值。
原文 arXiv:2004.02984;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.02984v2