Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Patrick Lewis‡†{}^{\dagger}{}^{\ddagger}, Pontus Stenetorp‡, Sebastian Riedel‡†{}^{\dagger}{}^{\ddagger} †Facebook AI Research; ‡University College London
Abstract
Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can actually generalize, and what drives their overall performance. We find that all models perform dramatically worse on questions that cannot be memorized from training sets, with a mean absolute performance difference of 63% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models outpe
中文速览
当前主流的开放域问答（Open-Domain QA）模型在标准测试集上分数亮眼，但这些分数究竟反映的是真正的泛化能力，还是对训练数据的死记硬背，一直缺乏细致分析。研究者对三个常用基准数据集（WebQuestions、TriviaQA、Open Natural Questions）进行了深入拆解，发现测试集中有60–70%的答案在训练集里出现过，另有约30%的测试问题在训练集中存在近似重复的换句表达（paraphrase）——这意味着大量"测试题"其实只考察模型的记忆力。基于这一发现，研究者将测试集按"问题重复""答案重复"和"全新问答"三类分别评估多个主流模型，结果显示所有模型在无法从训练集中记忆的问题上表现大幅下滑，平均绝对差距高达63%，而一个简单的最近邻检索模型甚至超过了BART闭卷问答模型。这项工作揭示了现有ODQA基准存在严重的数据泄露和记忆偏差问题，提醒研究社区在解读模型性能时必须区分"记忆"与"理解"，也为构建更可靠的评测基准指明了方向。
原文 arXiv:2008.02637;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2008.02637v1