TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
Wenhao Lu Bing Ads of AI、Research GroupMicrosoftOne Microsoft WayRedmondWA98052-6399 , Jian Jiao Bing Ads of AI、Research GroupMicrosoftOne Microsoft WayRedmondWA98052-6399 and Ruofei Zhang Bing Ads of AI、Research GroupMicrosoftOne Microsoft WayRedmondWA98052-6399
Abstract
Pre-trained language models like BERT have achieved great success in a wide variety of NLP tasks, while the superior performance comes with high demand in computational resources, which hinders the application in low-latency IR systems. We present TwinBERT model for effective and efficient retrieval, which has twin-structured BERT-like encoders to represent query and document respectively and a crossing layer to combine the embeddings and produce a similarity score. Different from BERT, where the two input sentences are concatenated and encoded together, TwinBERT decouples them during encoding and produces the embeddings for query and document independently, which allows document embeddings to be pre-computed offline and cached in memory. Thereupon, the computation left for run-time is from the query encoding and query-document crossing only. This single change can save large amount of computation time and resources, and therefore significantly improve serving efficiency. Moreover, a few well-designed network layers and training strategies are proposed to further reduce computational cost while at the same time keep the performance as remarkable as BERT model. Lastly, we develop tw
中文速览
搜索系统迫切需要既准确又快速的语义匹配模型，但 BERT 把查询和文档拼在一起编码，每次都要重算，延迟远超生产环境要求。TwinBERT 的核心思路是用两个独立的 BERT 式编码器分别为查询和关键词（文档）生成向量表示，再用一个轻量的交叉层合并打分，这样文档向量可以离线预计算并缓存，线上只需编码查询并做一次向量交叉，把推理复杂度从近似平方级降到线性级。配合三元字母（tri-letter）词嵌入、加权平均池化以及师生（teacher-student）知识蒸馏训练，模型在保持接近 BERT-Base 精度的同时，CPU 上对 100 个候选文档的打分延迟首次压缩到约 20 毫秒以内。该模型已集成进真实付费搜索系统，在几乎不影响线上延迟的前提下显著提升了相关性指标，为在大规模低延迟信息检索场景中落地预训练语言模型提供了切实可行的方案。
原文 arXiv:2002.06275;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.06275v1