What Makes Reading Comprehension Questions Easier?
Saku Sugawara♠, Kentaro Inui♣♢, Satoshi Sekine♢, Akiko Aizawa♡♠ ♠The University of Tokyo, ♣Tohoku University, ♢RIKEN AIP, ♡National Institute of Informatics
Abstract
A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.
中文速览
机器阅读理解（MRC）模型的真实能力究竟有多强，一直缺乏系统性检验——许多问题其实靠识别实体类型或词语匹配就能答对，根本不需要深层理解。作者对近年12个MRC数据集提出两种简单启发式规则：一是只输入问题的前k个词看模型还能否答对（测试实体类型识别偏差），二是检验正确答案是否就藏在与问题词汇最相似的那句话里（测试注意力匹配偏差），据此将每个数据集拆分为"容易"和"困难"子集，并让两个主流神经网络基线模型分别作答，再对样本问题进行人工标注，记录其有效性和所需推理技能。结果发现，模型在"困难"子集上的得分大幅下滑，困难问题普遍需要知识推理和跨句推理，而多项选择题比抽取式和描述式问题要求更广泛的推理能力，且歧义更少。这表明学界很可能高估了现有MRC系统的进步，数据集构建时若不主动过滤这类简单线索，评测结论将存在严重偏差。
原文 arXiv:1808.09384;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1808.09384v1