Simplified TinyBERT: Knowledge Distillation for Document Retrieval
Xuanang Chen(🖂) 1122 Ben He(🖂) 1122 Kai Hui This work has been done before joining Amazon.33 Le Sun 22 Yingfei Sun(🖂) 11
Abstract
Despite the effectiveness of utilizing the BERT model for document ranking, the high computational cost of such approaches limits their uses. To this end, this paper first empirically investigates the effectiveness of two knowledge distillation models on the document ranking task. In addition, on top of the recently proposed TinyBERT model, two simplifications are proposed. Evaluations on two different and widely-used benchmarks demonstrate that Simplified TinyBERT with the proposed simplifications not only boosts TinyBERT, but also significantly outperforms BERT-Base when providing 15 $\times$ speedup.
中文速览
用BERT做文档重排序效果很好，但推理速度慢、计算成本高，限制了实际部署。为此，研究者首先系统验证了标准知识蒸馏（Knowledge Distillation）和TinyBERT两种方法在文档排序任务上的有效性，随后在TinyBERT基础上提出两项改进：将任务特定阶段的两步训练合并为一步，并在损失函数中加入硬标签监督。在MS MARCO和TREC 2019 DL两个主流基准上的实验表明，改进后的Simplified TinyBERT不仅超越了原版TinyBERT，在推理速度提升15倍的条件下还能显著超过参数量更大的BERT-Base教师模型，同时将训练时间缩短约42%至52%。这项工作首次将知识蒸馏系统引入文档排序场景，为在实际搜索系统中以低成本部署高效排序模型提供了可行方案。
原文 arXiv:2009.07531;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.07531v2