XCMRC: Evaluating Cross-lingual Machine Reading Comprehension
Pengyuan Liu 11 Yuning Deng 1122 Chenghao Zhu 11 Han Hu 11
Abstract
We present XCMRC, the first public cross-lingual language understanding (XLU) benchmark which aims to test machines on their cross-lingual reading comprehension ability. To be specific, XCMRC is a Cross-lingual Cloze-style Machine Reading Comprehension task which requires the reader to fill in a missing word (we additionally provide ten noun candidates) in a sentence written in target language (English / Chinese) by reading a given passage written in source language (Chinese / English). Chinese and English are rich-resource language pairs, in order to study low-resource cross-lingual machine reading comprehension (XMRC), besides defining the common XCMRC task which has no restrictions on use of external language resources, we also define the pseudo low-resource XCMRC task by limiting the language resources to be used. In addition, we provide two baselines for common XCMRC task and two for pseudo XCMRC task respectively. We also provide an upper bound baseline for both tasks. We found that for common XCMRC task, translation-based method and multilingual sentence encoder-based method can obtain reasonable performance but still have much room for improvement. As for pseudo low-resourc
中文速览
跨语言阅读理解长期缺乏公开基准，研究者提出了XCMRC——首个跨语言完形填空式机器阅读理解（Cross-lingual Cloze-style Machine Reading Comprehension）数据集，要求模型读一段用源语言（中文或英文）写的文章，再从十个候选词中选出正确答案填入目标语言（英文或中文）的问题中。数据集从中英双语对照文章自动生成，共包含约11.4万条样本，并额外定义了"伪低资源"任务场景，通过限制可用语言资源来模拟真实低资源语言的挑战。实验评估了翻译式、多语言句子编码器（基于多语言BERT微调）、朴素直接迁移和段落无关四种基线方法，结果显示前两种方法在常规跨语言任务上能取得一定效果，但与单语言上界仍有显著差距，低资源场景下的方法则差距更大。这项工作填补了跨语言理解评测在阅读理解领域的空白，为推动中英及更广泛低资源语言的跨语言理解研究提供了重要的数据和基准。
原文 arXiv:1908.05416;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1908.05416v1