Adversarial NLI: A New Benchmark for Natural Language Understanding
Yixin Nie∗, Adina Williams†, Emily Dinan†, Mohit Bansal∗, Jason Weston†, Douwe Kiela† ∗UNC Chapel Hill †Facebook AI Research
Abstract
We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.
中文速览
自然语言推理（NLI）领域的顶尖模型正在被一个个静态数据集"喂饱"，测出来的高分并不真正代表理解能力。为此，研究者提出了一种名为 HAMLET 的"人机对抗循环"数据收集框架：让众包标注者针对当前最强模型的弱点，主动写出能让模型判断出错的样本，通过人工验证后加入训练集、训练更强的模型，再重复这一过程，共迭代三轮，最终构建出对抗性自然语言推理数据集 ANLI（Adversarial NLI）。实验表明，用 ANLI 训练后的模型不仅在 SNLI、MNLI 等主流基准上达到了最先进水平，在更难的 ANLI 测试集上也明显优于原有基线，而即便是非专业标注者仍能以相当高的成功率欺骗最强模型集成。这项工作的重要性在于：它提供了一个可持续演进的"动态靶标"，而不是一个很快就会饱和的固定基准，从而让评估真正追得上模型进步的速度，同时也揭示了现有模型依赖虚假统计规律而非真正语言理解的根本缺陷。
原文 arXiv:1910.14599;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.14599v2