Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao†, Shaohan Huang‡, Wenhui Wang‡, Li Dong‡, Furu Wei‡ †Shandong University, Jinan, China ‡Microsoft Research, Beijing, China Contribution during internship at Microsoft Research.
Abstract
Large pretrained models have achieved great success in many natural language processing tasks. However, when they are applied in specific domains, these models suffer from domain shift and bring challenges in fine-tuning and online serving for latency and capacity constraints. In this paper, we present a general approach to developing small, fast and effective pretrained models for specific domains. This is achieved by adapting the off-the-shelf general pretrained models and performing task-agnostic knowledge distillation in target domains. Specifically, we propose domain-specific vocabulary expansion in the adaptation stage and employ corpus level occurrence probability to choose the size of incremental vocabulary automatically. Then we systematically explore different strategies to compress the large pretrained models for specific domains. We conduct our experiments in the biomedical and computer science domain. The experimental results demonstrate that our approach achieves better performance over the $\text{BERT}_{\text{BASE}}$ model in domain-specific tasks while 3.3× smaller and 5.1× faster than $\text{BERT}_{\text{BASE}}$ . The code and pretrained models are available at htt
中文速览
大型预训练语言模型在特定专业领域（如生物医学、计算机科学）应用时，既面临领域偏移导致的性能下降，又因参数量庞大难以在边缘设备上部署。AdaLM 框架通过两步走解决这一问题：先用基于语料出现概率的自动词表扩展将通用模型适配到目标领域，再系统比较四种"领域适配"与"知识蒸馏"的组合策略，最终发现"先分别对大小模型做领域适配，再以两者为师生对进行蒸馏"的 Adapt-and-Distill 方案效果最佳。实验结果显示，该方案得到的6层小模型在生物医学和计算机科学领域的下游任务上超越了 BERT-BASE，同时体积缩小 3.3 倍、推理速度提升 5.1 倍。这项工作为在资源受限场景下高效部署领域专用预训练模型提供了一套系统性、可复用的通用框架。
原文 arXiv:2106.13474;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.13474v2