Quasar: Datasets for Question Answering by Search and Reading
Bhuwan Dhingra Kathryn Mazaitis William W. Cohen School of Computer Science Carnegie Mellon University {bdhingra, krivard,
Abstract
We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 (Callan et al., 2009) serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and
中文速览
现有阅读理解数据集大多假设相关段落已知，无法评估"先检索再理解"这一完整的开放域问答流程，研究者们因此缺乏能同时考察检索与阅读两个环节的大规模基准。为此，作者构建了两个新数据集 Quasar：Quasar-S 包含约 3.7 万道从 Stack Overflow 标签定义生成的完形填空题，以该网站的帖子作为语料库；Quasar-T 包含约 4.3 万道开放域常识问答题，以 ClueWeb09 约十亿网页为检索背景，并随数据集一并发布了预检索的相关文档，方便研究者单独研究阅读理解部分。实验结果显示，从简单启发式方法到当前最强神经网络模型，最佳自动系统在两个数据集上的准确率分别比人类低 16.4% 和 32.1%，表明现有方法在"检索覆盖率"与"阅读精准度"之间仍存在明显瓶颈。这两个数据集的价值在于推动社区研究如何将检索与阅读联合训练，以构建真正端到端的问答系统。
原文 arXiv:1707.03904;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.03904v2