Testing the Generalization Power of Neural Network Models Across NLI Benchmarks
Aarne Talman Department of Digital Humanities University of Helsinki Basement AI \AndStergios Chatzikyriakidis Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg Basement AI
Abstract
Neural network models have been very successful in natural language inference, with the best models reaching 90% accuracy in some benchmarks. However, the success of these models turns out to be largely benchmark specific. We show that models trained on a natural language inference dataset drawn from one benchmark fail to perform well in others, even if the notion of inference assumed in these benchmarks is the same or similar. We train six high performing neural network models on different datasets and show that each one of these has problems of generalizing when we replace the original test set with a test set taken from another corpus designed for the same task. In light of these results, we argue that most of the current neural network models are not able to generalize well in the task of natural language inference. We find that using large pre-trained language models helps with transfer learning when the datasets are similar enough. Our results also highlight that the current NLI datasets do not cover the different nuances of inference extensively enough.
中文速览
当前最优的自然语言推理（Natural Language Inference, NLI）神经网络模型在各自的基准测试集上能达到90%左右的准确率，但这种高分究竟反映的是真正的推理能力，还是只是对特定数据集的记忆？研究者选取了六种主流神经网络模型，分别在SNLI、MultiNLI、SICK三个NLI数据集上训练，然后交叉测试——即用在A数据集上训练好的模型去预测B数据集的测试集。结果发现，跨数据集时模型的准确率平均下降约25个百分点，即便是标注方式和推理定义几乎相同的SNLI与MultiNLI之间，迁移效果依然很差；唯有基于大规模预训练语言模型的BERT在相似数据集间表现出了相对较强的迁移能力，但面对构造方式差异较大的SICK时同样大幅失准。这项研究揭示了一个严峻的现实：现有NLI模型学到的主要是数据集特有的统计模式，而非通用的语言推理规律，这既提示我们需要构建更多样、更全面的NLI数据集，也警示整个NLP领域不能将基准测试分数等同于真实的语言理解能力。
原文 arXiv:1810.09774;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.09774v3