Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord Allen Institute for Artificial Intelligence, Seattle, WA, U.S.A.
Abstract
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
中文速览
为了推动AI在真正困难的问答任务上取得突破，研究者构建了一个名为ARC（AI2 Reasoning Challenge）的基准数据集，包含7787道来自真实标准化考试的小学科学选择题，并特别筛选出2590道"挑战题"——这些题目同时难倒了基于信息检索和词共现统计的两种简单算法，因此必须依赖真正的知识推理才能作答。研究团队还配套发布了一个含1400万条科学相关句子的语料库，以及三个改编自当时顶尖神经网络模型（DecompAttn、BiDAF、DGEM）的基线系统。测试结果表明，这些领先模型在挑战题集上的表现与随机猜答无显著差异，凸显了该任务的真实难度。ARC的意义在于，它通过主动过滤掉能被简单方法解决的题目，将研究压力精准导向需要常识、多步推理等高阶能力的问题，从而避免AI进步被低估或被简单刷分所掩盖。
原文 arXiv:1803.05457;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.05457v1