Large-scale Cloze Test Dataset Created by Teachers
Qizhe Xie , Guokun Lai , Zihang Dai, Eduard Hovy Language Technologies Institute, Carnegie Melon University {qizhex, guokun, dzihang, Equal contribution.
Abstract
Cloze tests are widely adopted in language exams to evaluate students’ language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH 111CLOTH (CLOze test by TeacHers) is available at http://www.cs.cmu.edu/~glai1/data/cloth/. 222The leaderboard is available at http://www.qizhexie.com/data/CLOTH_leaderboard.html, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.
中文速览
完形填空（cloze test）长期是语言水平考试的核心题型，但现有机器阅读理解研究所用的完形填空数据集大多由程序自动生成，题目质量参差不齐、难度普遍偏低，导致模型很快就追上了人类水准。为此，研究者从中国中学英语考试中收集了超过九万九千道由教师人工命题的完形填空题，构建了大规模数据集CLOTH，这些题目经过精心设计，干扰选项高度相近，需要理解长距离语境才能作答。实验表明，即使是在十亿词语料上训练的顶尖语言模型，也与人类表现存在显著差距；进一步分析发现，模型的主要瓶颈在于无法有效利用跨句子的长程上下文信息——当人类被限制为只能看单句时，其准确率便与模型相当。CLOTH为语言建模和机器阅读理解两个领域提供了一个更贴近真实考试、更能检验深层语言理解能力的高质量评测平台。
原文 arXiv:1711.03225;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.03225v3