Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline
Luyu Gao Zhuyun Dai Jamie Callan
Abstract
Pre-trained deep language models (LM) have advanced the state-of-the-art of text retrieval. Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextualized matching signals. Meanwhile, deep LMs can also be leveraged to improve search index, building retrievers with better recall. One would expect a straightforward combination of both in a pipeline to have additive performance gain. In this paper, we discover otherwise and that popular reranker cannot fully exploit the improved retrieval result. We, therefore, propose a Localized Contrastive Estimation (LCE) for training rerankers and demonstrate it significantly improves deep two-stage models.111Our codes are open sourced at https://github.com/luyug/Reranker.
中文速览
把更强的第一阶段检索器（如用BERT重新估计词权重的HDCT）与BERT重排序器直接拼接，并不能保证最终排名同步提升——因为候选列表变难后，那些"看起来很像相关文档"的干扰负例会让用传统二元交叉熵训练的重排序器判断失误。为此，研究者提出了"局部对比估计"（Localized Contrastive Estimation，LCE）训练方法：从目标检索器返回的Top候选中采样负例，同时采用对比损失让模型在区分正例时必须同时压制多个难负例，从而避免模型依赖干扰特征短路。在MSMARCO文档排序任务上，LCE在不增加训练和推理开销的前提下，显著超越了相同BERT模型配合传统训练方式的基线，检索器越强时提升越大，单模型与集成模型均达到当时最优水平。这项工作说明，提升流水线检索系统性能不必依赖更大的预训练模型，改进训练目标本身就能充分释放现有模型的潜力，同时也为社区系统性研究检索器与重排序器之间的交互关系提供了新视角。
原文 arXiv:2101.08751;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.08751v1