QASC: A Dataset for Question Answering via Sentence Composition
Tushar Khot Peter Clark Michal Guerquin Peter Jansen+ Ashish Sabharwal Allen Institute for AI, Seattle, WA, U.S.A. +University of Arizona, Tucson, AZ, U.S.A.
Abstract
Composing knowledge from multiple pieces of texts is a key challenge in multi-hop question answering. We present a multi-hop reasoning dataset, Question Answering via Sentence Composition (QASC), that requires retrieving facts from a large corpus and composing them to answer a multiple-choice question. QASC is the first dataset to offer two desirable properties: (a) the facts to be composed are annotated in a large corpus, and (b) the decomposition into these facts is not evident from the question itself. The latter makes retrieval challenging as the system must introduce new concepts or relations in order to discover potential decompositions. Further, the reasoning model must then learn to identify valid compositions of these retrieved facts using common-sense reasoning. To help address these challenges, we provide annotation for supporting facts as well as their composition. Guided by these annotations, we present a two-step approach to mitigate the retrieval challenges. We use other multiple-choice datasets as additional training data to strengthen the reasoning model. Our proposed approach improves over current state-of-the-art language models by 11% (absolute). The reasoning a
中文速览
多跳问答（multi-hop QA）要求模型把来自不同文本的知识片段拼合起来推理出答案，而现有数据集要么缺少标注、要么问题结构太透明——光看问题就能猜到该怎么拆解，这让模型很难真正学会"组合新知识"这项核心能力。为此，研究团队构建了 QASC（Question Answering via Sentence Composition）数据集，包含约一万道选择题，每道题都需要从一个拥有1700万条句子的大型语料库中检索两条事实并加以组合才能作答，且题目本身不会透露应如何分解关系——模型必须自己"脑补"中间概念和隐含关系。研究者还为每道题标注了两条支撑事实及其组合方式，并在此基础上提出了一种两步检索方法：先用问题检索第一条事实，再用该事实引导检索第二条，将黄金事实的召回率提升了43个百分点、答题准确率提升了14个百分点；配合大规模预训练语言模型和额外10万条外部训练数据，最终系统比此前最优方法高出11%（绝对值），但距人类93%的水平仍有20%的差距，说明隐式关系的检索与组合推理依然是尚未解决的难题。
原文 arXiv:1910.11473;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.11473v2