Who did What: A Large-Scale Person-Centered Cloze Dataset
Takeshi Onishi Hai Wang Mohit Bansal Kevin Gimpel David McAllester Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA to appear at EMNLP 2016.
Abstract
We have constructed a new “Who-did-What” dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus. The WDW dataset has a variety of novel features. First, in contrast with the CNN and Daily Mail datasets [Hermann et al., 2015] we avoid using article summaries for question formation. Instead, each problem is formed from two independent articles — an article given as the passage to be read and a separate article on the same events used to form the question. Second, we avoid anonymization — each choice is a person named entity. Third, the problems have been filtered to remove a fraction that are easily solved by simple baselines, while remaining 84% solvable by humans. We report performance benchmarks of standard systems and propose the WDW dataset as a challenge task for the community.111Available at tticnlp.github.io/who_did_what
中文速览
为了推动机器阅读理解研究，研究者从英文新闻语料库Gigaword构建了一个名为"Who-did-What"（WDW）的大规模完形填空数据集，包含超过20万道多项选择题。与CNN/Daily Mail等已有数据集不同，WDW的每道题由两篇独立文章组成——一篇作为阅读材料，另一篇用于生成问题，答案选项均为人名实体且不做匿名化处理，这使得问题与文章之间的语言表面相似度更低，对语义理解的要求更高。为了剔除靠简单规则就能答对的题目，研究者还专门设计了基线抑制机制，过滤掉可被"选最常出现的人"或"选第一个出现的人"等基线轻松解决的题目。实验结果显示，人类在该数据集上的准确率达84%，而现有最优模型仅在59%左右，机器与人类之间的差距明显大于CNN等数据集，表明WDW是一个更具挑战性的阅读理解基准，能更有效地衡量模型的深层语义理解能力。
原文 arXiv:1608.05457;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.05457v1