RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
Yingqi Qu1, Yuchen Ding1, Jing Liu1, Kai Liu1, Ruiyang Ren2 Wayne Xin Zhao, Daxiang Dong1, Hua Wu1 and Haifeng Wang1 1Baidu Inc.; 2Gaoling School of Artificial Intelligence, Renmin University of China {quyingqi, dingyuchen, liujing46, liukai20, dongdaxiang, wu_hua, Corresponding authors. † The work was done when Ruiyang Ren was doing internship at Baidu.
Abstract
In open-domain question answering, dense passage retrieval has become a new paradigm to retrieve relevant passages for finding answers. Typically, the dual-encoder architecture is adopted to learn dense representations of questions and passages for semantic matching. However, it is difficult to effectively train a dual-encoder due to the challenges including the discrepancy between training and inference, the existence of unlabeled positives and limited training data. To address these challenges, we propose an optimized training approach, called RocketQA, to improving dense passage retrieval. We make three major technical contributions in RocketQA, namely cross-batch negatives, denoised hard negatives and data augmentation. The experiment results show that RocketQA significantly outperforms previous state-of-the-art models on both MSMARCO and Natural Questions. We also conduct extensive experiments to examine the effectiveness of the three strategies in RocketQA. Besides, we demonstrate that the performance of end-to-end QA can be improved based on our RocketQA retriever 111Our code is available at https://github.com/PaddlePaddle/Research/tree/master/NLP/NAACL2021-RocketQA.
中文速览
开放域问答系统需要先从海量文档中检索出相关段落，双编码器（dual-encoder）架构虽然是主流方案，但训练时面临三大难题：训练与推理阶段的候选规模差距悬殊、标注数据中存在大量漏标的正样本（unlabeled positives）导致难负样本质量差，以及有监督训练数据总量不足。针对这三个问题，研究者提出了 RocketQA 训练框架，核心包含三项技术：跨批次负样本（cross-batch negatives）通过多 GPU 共享段落表示来大幅增加每个问题的可用负样本数量；去噪难负样本（denoised hard negatives）借助能力更强的交叉编码器（cross-encoder）过滤掉排名靠前但实为正样本的假负例；数据增强（data augmentation）则用交叉编码器为海量无标注数据打上伪标签，再用来训练双编码器。实验结果表明，RocketQA 在 MSMARCO 和 Natural Questions 两个权威基准上均显著超越此前最优模型，并进一步提升了端到端问答的整体性能。这项工作系统性地解决了密集段落检索（dense passage retrieval）训练中长期存在的核心瓶颈，为开放域问答系统提供了更高质量的检索基础。
原文 arXiv:2010.08191;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.08191v2