Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova Equal contribution. Ordering decided by coin toss. Jimmy Lin University of Waterloo {r33tang, yao.lu, {ovechtom,
Abstract
In the natural language processing literature, neural networks are becoming increasingly deeper and complex. The recent poster child of this trend is the deep language representation model, which includes BERT, ELMo, and GPT. These developments have led to the conviction that previous-generation, shallower neural networks for language understanding are obsolete. In this paper, however, we demonstrate that rudimentary, lightweight neural networks can still be made competitive without architecture changes, external training data, or additional input features. We propose to distill knowledge from BERT, a state-of-the-art language representation model, into a single-layer BiLSTM, as well as its siamese counterpart for sentence-pair tasks. Across multiple datasets in paraphrasing, natural language inference, and sentiment classification, we achieve comparable results with ELMo, while using roughly 100 times fewer parameters and 15 times less inference time.
中文速览
深度语言模型如BERT虽然效果强大，但参数量巨大、推理速度慢，难以部署在移动端或实时系统中，而人们普遍认为更简单的旧式神经网络已经过时。本文提出用知识蒸馏（knowledge distillation）的方法，把BERT学到的"软知识"（logit输出）迁移给一个单层双向长短期记忆网络（BiLSTM），让这个轻量小模型去模仿BERT的预测行为；为了让蒸馏更有效，作者还设计了一套基于规则的文本数据增强方法，通过随机遮词、同词性词替换和n-gram采样来扩充无标注训练数据。在释义检测、自然语言推理和情感分类等多个任务上，蒸馏后的BiLSTM达到了与ELMo相当的性能，参数量却少约100倍、推理速度快约15倍。这项工作说明简单网络并非真的过时，通过合理的知识迁移完全可以在资源受限场景下兼顾效果与效率。
原文 arXiv:1903.12136;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.12136v1