A Replication Study of Dense Passage Retriever
Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo
Abstract
Text retrieval using learned dense representations has recently emerged as a promising alternative to “traditional” text retrieval using sparse bag-of-words representations. One recent work that has garnered much attention is the dense passage retriever (DPR) technique proposed by Karpukhin et al. (2020) for end-to-end open-domain question answering. We present a replication study of this work, starting with model checkpoints provided by the authors, but otherwise from an independent implementation in our group’s Pyserini IR toolkit and PyGaggle neural text ranking library. Although our experimental results largely verify the claims of the original paper, we arrived at two important additional findings that contribute to a better understanding of DPR: First, it appears that the original authors under-report the effectiveness of the BM25 baseline and hence also dense–sparse hybrid retrieval results. Second, by incorporating evidence from the retriever and an improved answer span scoring technique, we are able to improve end-to-end question answering effectiveness using exactly the same models as in the original work.
中文速览
密集段落检索（Dense Passage Retrieval，DPR）是近年来备受关注的一种用神经网络学习稠密向量来做文本检索的方法，被应用于开放域问答任务，但原始论文的结论是否可靠、有没有改进空间仍有待验证。研究团队基于原作者发布的模型权重，用自己独立开发的工具包重新实现了整套流程，对DPR进行了严格的复现研究。复现结果总体印证了原论文的结论，但同时发现了两处重要偏差：原论文低估了传统稀疏检索方法BM25的基线水平，导致其认为稠密与稀疏检索融合（hybrid retrieval）没有显著收益，而实验显示融合方法在五个标准数据集上平均可提升约三个百分点的top-20检索准确率；此外，通过引入检索器打分并采用改进的答案片段打分策略，在不改变任何模型参数的前提下，端到端问答的精确匹配（Exact Match）得分也能再提升约三个百分点。这项工作说明，即便是已被广泛引用的重要论文也可能存在基线低估和结果改进空间，严格的复现研究对推动领域正确认知具有不可忽视的价值。
原文 arXiv:2104.05740;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.05740v1