MicroNet for Efficient Language Modeling
\NameZhongxia Yan \addrMassachusetts Institute of Technology \NameHanrui Wang \addrMassachusetts Institute of Technology \NameDemi Guo \addrHarvard University \NameSong Han \addrMassachusetts Institute of Technology
Abstract
It is important to design compact language models for efficient deployment. We improve upon recent advances in both the language modeling domain and the model-compression domain to construct parameter and computation efficient language models. We use an efficient transformer-based architecture with adaptive embedding and softmax, differentiable non-parametric cache, Hebbian softmax, knowledge distillation, network pruning, and low-bit quantization. In this paper, we provide the winning solution to the NeurIPS 2019 MicroNet Challenge in the language modeling track. Compared to the baseline language model provided by the MicroNet Challenge, our model is 90 times more parameter-efficient and 36 times more computation-efficient while achieving the required test perplexity of 35 on the Wikitext-103 dataset. We hope that this work will aid future research into efficient language models, and we have released our full source code on GitHub.
中文速览
为了让语言模型能在资源受限的设备上高效运行，研究者将语言建模领域和模型压缩领域的多项技术融合在一起，构建出既省参数又省计算量的紧凑型语言模型。具体方法包括：采用带自适应词嵌入与softmax的Transformer-XL架构、可微非参数缓存（differentiable non-parametric cache）、Hebbian softmax，以及知识蒸馏（knowledge distillation）、网络剪枝（pruning）和低比特量化（quantization）等压缩手段。最终模型在Wikitext-103数据集上以35的测试困惑度（perplexity）达标，与MicroNet挑战赛官方基准模型相比，参数量减少90倍、计算量减少36倍，并斩获NeurIPS 2019 MicroNet挑战赛语言建模赛道冠军。这项工作证明了将多种压缩与建模技术系统性叠加可以极大突破语言模型的效率边界，为未来在边缘设备上部署自然语言处理模型提供了重要参考。
原文 arXiv:2005.07877;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.07877v1