A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Adina Williams1、Nikita Nangia2、Samuel R. Bowman1,2,3 \AND1Department of Linguistics New York University、2Center for Data Science New York University、3Department of Computer Science New York University
Abstract
This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.
中文速览
自然语言推理（Natural Language Inference，NLI）任务要求模型判断两个句子之间是蕴含、矛盾还是中立关系，是评估机器语言理解能力的理想基准，但此前最主流的 SNLI 数据集只来源于图片说明这单一体裁，导致语言现象单调、难度偏低、顶尖模型已接近人类水平。为此，研究者构建了 MultiNLI——一个包含 43.3 万对句子的大规模 NLI 数据集，素材横跨面对面对话、政府报告、小说、电话录音、旅游指南等十种英语体裁，并特意让其中五种体裁只出现在测试集而非训练集，从而同时支持"同领域"和"跨领域迁移"两种评估场景。实验表明，现有在 SNLI 上表现优异的模型在 MultiNLI 上准确率显著下降，说明该数据集难度更高、区分度更强，同时标注一致性与 SNLI 持平，保证了数据质量。MultiNLI 的发布为机器自然语言理解研究提供了一个更全面、更具挑战性的公开基准，也为探索跨领域泛化能力提供了直接的实验平台。
原文 arXiv:1704.05426;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.05426v4