Stress-Testing Neural Models of Natural Language Inference with Multiply-Quantified Sentences
Atticus Geiger Stanford Symbolic Systems Program Lauri Karttunen Stanford Linguistics、Ignacio Cases Stanford Linguistics Christopher Potts Stanford Linguistics
Abstract
Standard evaluations of deep learning models for semantics using naturalistic corpora are limited in what they can tell us about the fidelity of the learned representations, because the corpora rarely come with good measures of semantic complexity. To overcome this limitation, we present a method for generating data sets of multiply-quantified natural language inference (NLI) examples in which semantic complexity can be precisely characterized, and we use this method to show that a variety of common architectures for NLI inevitably fail to encode crucial information; only a model with forced lexical alignments avoids this damaging information loss.
中文速览
想搞清楚深度学习模型到底有没有真正学到语言的语义，光靠自然语料库是不够的，因为这些语料库本身缺乏对语义复杂度的精确刻画。研究者提出了一种自动生成自然语言推理（NLI）数据集的方法，通过形式语法生成含多个量词、修饰语和否定的句子，并用定理证明器自动标注蕴含关系，从而精确控制每个样本的语义复杂度。实验发现，LSTM、树形神经网络（TreeNN）等主流架构都只能学到约96%的准确率，并陷入同一种系统性错误：它们能学会量词和否定词的逻辑关系，却无法保留句子中具体实义词的身份信息，根本原因在于将前提和假设分别压缩成独立向量表示所造成的信息瓶颈；只有强制对齐前提与假设中对应词汇的组合树模型（CompTreeNN）才能完美完成任务。这一结果表明，当前主流的深度语义表示架构存在根本性的结构缺陷，对于依赖精确语义组合的任务而言，仅靠提升维度或调整超参数都无法解决问题。
原文 arXiv:1810.13033;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.13033v1