Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently
Jingtao Zhan BNRist, DCST, Tsinghua University Beijing, China \AndJiaxin Mao BNRist, DCST, Tsinghua University Beijing, China \AndYiqun Liu BNRist, DCST, Tsinghua University Beijing, China \AndMin Zhang BNRist, DCST, Tsinghua University Beijing, China \AndShaoping Ma BNRist, DCST, Tsinghua University Beijing, China Corresponding author
Abstract
Ranking has always been one of the top concerns in information retrieval research. For decades, lexical matching signal has dominated the ad-hoc retrieval process, but it also has inherent defects, such as the vocabulary mismatch problem. Recently, Dense Retrieval (DR) technique has been proposed to alleviate these limitations by capturing the deep semantic relationship between queries and documents. The training of most existing Dense Retrieval models relies on sampling negative instances from the corpus to optimize a pairwise loss function. Through investigation, we find that this kind of training strategy is biased and fails to optimize full retrieval performance effectively and efficiently. To solve this problem, we propose a Learning To Retrieve (LTRe) training technique. LTRe constructs the document index beforehand. At each training iteration, it performs full retrieval without negative sampling and then updates the query representation model parameters. Through this process, it teaches the DR model how to retrieve relevant documents from the entire corpus instead of how to rerank a potentially biased sample of documents. Experiments in both passage retrieval and document re
中文速览
信息检索中的稠密检索（Dense Retrieval）技术能让模型理解查询与文档的语义关系，但现有训练方法普遍依赖"负样本采样"——从语料库中挑出一批文档让模型学会重排，而真正推理时模型却要从整个语料库中直接检索，两者之间存在根本性的训练-推理不一致。针对这一问题，作者提出了 LTRe（Learning To Retrieve）训练机制：在训练前预先用固定的文档编码器构建好完整文档索引，每次迭代都直接对全量语料库做检索，再根据检索结果反向更新查询编码器参数，让模型真正学会"如何从整库中找相关文档"而非"如何在有偏的小样本中重排"。实验结果表明，LTRe 在段落和文档检索任务上显著超越所有稀疏与稠密基线，在合理延迟约束下甚至优于 BM25+BERT 级联系统，同时与当时最优的 ANCE 方法相比训练速度提升逾 170 倍，使用压缩索引还能将所需 GPU 从 4 块降到 1 块。这项工作不仅在效果上刷新了第一阶段检索的标杆，更将稠密检索的训练开销压缩到实用级别，具有重要的工程落地价值。
原文 arXiv:2010.10469;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.10469v1