Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Alex Warstadt1、Samuel R. Bowman1,2,3 \AND1Dept. of Linguistics New York University 10 Washington Place New York, NY 10003、2Dept. of Computer Science New York University 60 Fifth Avenue New York, NY 10011、3Center for Data Science New York University 60 Fifth Avenue New York, NY 10011
Abstract
Recent work on evaluating grammatical knowledge in pretrained sentence encoders gives a fine-grained view of a small number of phenomena. We introduce a new analysis dataset that also has broad coverage of linguistic phenomena. We annotate the development set of the Corpus of Linguistic Acceptability (CoLA; Warstadt et al., 2018) for the presence of 13 classes of syntactic phenomena including various forms of argument alternations, movement, and modification. We use this analysis set to investigate the grammatical knowledge of three pretrained encoders: BERT Devlin et al. (2018), GPT Radford et al. (2018), and the BiLSTM baseline from Warstadt et al. We find that these models have a strong command of complex or non-canonical argument structures like ditransitives (Sue gave Dan a book) and passives (The book was read). Sentences with long-distance dependencies like questions (What do you think I ate?) challenge all models, but for these, BERT and GPT have a distinct advantage over the baseline. We conclude that recent sentence encoders, despite showing near-human performance on acceptability classification overall, still fail to make fine-grained grammaticality distinctions for many
中文速览
预训练语言模型（如BERT、GPT）在句子可接受性判断任务上已接近人类水平，但我们并不清楚它们究竟掌握了哪些语法知识、又在哪些句法结构上仍然力不从心。作者对CoLA语料库的开发集全部1043个句子进行了专家级句法标注，覆盖13大类、59小类句法现象，构建了一个兼具细粒度和广覆盖的分析数据集，并以此为探针任务评估了BERT、GPT和BiLSTM基线模型的语法能力。结果显示，三类模型对被动句、双宾结构等非标准论元结构均有较强的掌握，但面对包含长距离依存关系的疑问句时普遍表现变差，而BERT和GPT在此类结构上明显优于BiLSTM。这项工作揭示了当前最先进编码器在整体性能接近人类的背后，对许多复杂句法结构的细粒度区分能力仍存在明显短板，为未来模型的改进和语言习得的理论研究提供了重要参考。
原文 arXiv:1901.03438;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.03438v4