A Span-Extraction Dataset for Chinese Machine Reading Comprehension
Yiming Cui†‡, Ting Liu†, Wanxiang Che†, Li Xiao‡, Zhipeng Chen‡, Wentao Ma‡§, Shijin Wang‡§, Guoping Hu‡ †Research Center for Social Computing and Information Retrieval (SCIR), Harbin Institute of Technology, Harbin, China ‡State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China §iFLYTEK AI Research (Hebei), Langfang, China
Abstract
Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, the existing reading comprehension datasets are mostly in English. In this paper, we introduce a Span-Extraction dataset for Chinese machine reading comprehension to add language diversities in this area. The dataset is composed by near 20,000 real questions annotated on Wikipedia paragraphs by human experts. We also annotated a challenge set which contains the questions that need comprehensive understanding and multi-sentence inference throughout the context. We present several baseline systems as well as anonymous submissions for demonstrating the difficulties in this dataset. With the release of the dataset, we hosted the Second Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2018). We hope the release of the dataset could further accelerate the Chinese machine reading comprehension research.111Resources are available: https://github.com/ymcui/cmrc2018.
中文速览
机器阅读理解（Machine Reading Comprehension）领域长期缺乏高质量的中文数据集，已有模型在英文SQuAD上表现优异，却难以直接迁移验证到中文场景。为此，作者构建了CMRC 2018——一个基于中文维基百科、由人工专家标注的片段抽取式（span-extraction）阅读理解数据集，包含近2万道问题，并额外制作了一个需要跨句多步推理的"挑战集"。实验结果显示，主流模型在常规测试集上F1可达80分以上，与人类表现差距仅约10个百分点，但在挑战集上F1骤降至50%以下，而人类表现几乎不受影响，表明现有模型在复杂推理方面仍有明显短板。这一数据集的发布不仅填补了中文阅读理解资源的空白，也为跨语言研究和推动更强推理能力的模型研发提供了重要基准。
原文 arXiv:1810.07366;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.07366v2