Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan and Allan Hanbury TU Wien, Vienna, Austria
Abstract
Retrieval and ranking models are the backbone of many applications such as web search, open domain QA, or text-based recommender systems. The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller student model. We find that different ranking architectures tend to produce output scores in different magnitudes. Based on this finding, we propose a cross-architecture training procedure with a margin focused loss (Margin-MSE), that adapts knowledge distillation to the varying score output distributions of different BERT and non-BERT passage ranking architectures. We apply the teachable information as additional fine-grained labels to existing training triples of the MSMARCO-Passage collection. We evaluate our procedure of distilling knowledge from state-
中文速览
不同神经排序架构（如BERT拼接模型、双编码器、ColBERT等）在查询延迟和检索效果之间存在明显取舍——高效架构跑得快，但效果往往不如慢而准的BERT拼接模型。为了弥补这一效果差距，作者提出了一种跨架构知识蒸馏（cross-architecture knowledge distillation）方法：关键洞察在于不同架构的输出分数分布差异悬殊，无法直接对齐，因此设计了基于"分数差值"的Margin-MSE损失函数，让高效的学生模型去拟合教师模型对相关与非相关文档打分之间的差距，而非绝对分值，从而绕开了不同架构评分尺度不一致的问题。在MSMARCO段落检索基准上，作者将由BERT-Base、BERT-Large和ALBERT-Large组成的教师集成模型的知识蒸馏到TK、ColBERT、PreTT和BERT点积模型四种高效架构中，结果表明所有学生模型的检索效果均显著提升，PreTT和ColBERT甚至超过了原本更慢的BERT拼接基线，BERT点积模型在无需昂贵负样本挖掘的情况下也达到了与专门训练方法相当的密集检索效果。这项工作为工业界在不牺牲查询效率的前提下大幅提升神经排序模型效果提供了一条实用路径，作者还公开了教师打分文件和代码以方便社区复用。
原文 arXiv:2010.02666;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.02666v2