BERT & Family Eat Word Salad: Experiments with Text Understanding
Ashim Gupta, Giorgi Kvernadze, Vivek Srikumar
Abstract
In this paper, we study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. We define simple heuristics to construct such examples. Our experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. As a consequence of this phenomenon, models trained on sentences with randomly permuted word order perform close to state-of-the-art models. To alleviate these issues, we show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.
中文速览
当前最强的BERT类自然语言处理模型，在面对被故意打乱成"文字沙拉"的无意义输入时，依然能以高达95%左右的置信度给出预测，完全无法识别输入已经失去意义。研究者设计了九种"破坏性变换"——如将句子单词随机打乱、按字母排序、或用统计机器翻译生成无语法乱序文本——来系统验证这一现象，并在自然语言推断、释义检测、情感分类三类任务上展开实验。结果发现，模型不仅对无意义输入给出与原始输入高度一致的预测，甚至用这些乱序无意义句子训练出来的模型，性能也接近在正常句子上训练的最优模型。这说明现有模型根本没有真正理解语言形式，而是依赖与语言结构无关的浅层统计线索；好消息是，若在训练时加入对无效输入的显式识别目标，模型可以在不损失原有性能的前提下，学会拒绝这类无意义输入。
原文 arXiv:2101.03453;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.03453v2