Towards Robust Neural Retrieval Models with Synthetic Pre-Training
Revanth Gangi Reddy1, Vikas Yadav2, Md Arafat Sultan2, Martin Franz2, Vittorio Castelli2, Heng Ji1, Avirup Sil2 1University of Illinois at Urbana-Champaign 2IBM Research AI, New York
Abstract
Recent work has shown that commonly available machine reading comprehension (MRC) datasets can be used to train high-performance neural information retrieval (IR) systems. However, the evaluation of neural IR has so far been limited to standard supervised learning settings, where they have outperformed traditional term matching baselines. We conduct in-domain and out-of-domain evaluations of neural IR, and seek to improve its robustness across different scenarios, including zero-shot settings. We show that synthetic training examples generated using a sequence-to-sequence generator can be effective towards this goal: in our experiments, pre-training with synthetic examples improves retrieval performance in both in-domain and out-of-domain evaluation on five different test sets.
中文速览
神经信息检索（neural IR）系统在标准监督场景下表现出色，但一旦换到与训练数据差异较大的领域，性能就会大幅下滑，甚至不如传统的BM25词频匹配方法。为此，研究者利用BART序列到序列模型，从Wikipedia文章中自动生成大量合成问答对，用这些合成数据对Dense Passage Retriever（DPR）进行预训练，再在真实标注数据上微调，得到增强版模型AugDPR。实验在五个数据集上验证，包括同领域、近领域（同为Wikipedia语料）和远领域（生物医学文献）的零样本测试，AugDPR在所有场景下均稳定超越原始DPR，即便在词汇重叠率仅17%的生物医学远领域，也显著缩小了与BM25的差距。这项工作表明，用合成数据预训练是提升神经检索模型跨域鲁棒性的有效且低成本的手段，为将神经IR推广到无标注新领域提供了可行路径。
原文 arXiv:2104.07800;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.07800v1