Assessing BERT’s Syntactic Abilities
Yoav Goldberg1,2 1 Computer Science Department, Bar Ilan University 2 Allen Institute for Artificial Intelligence ,
Abstract
I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) “coloreless green ideas” subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.
中文速览
大规模预训练语言模型BERT到底能不能理解英语句法结构，这个问题此前并不明朗。研究者借用三套已有的测试材料——真实维基百科句子、将实词随机替换后的"无色绿色想法"乱词句子、以及人工构造的最小对比句——来考察BERT在主谓一致和反身代词回指这两类经典句法现象上的表现，方法是将目标动词遮盖后让BERT预测正确与错误形式的得分高低。结果出乎意料：无需任何微调的预训练BERT在几乎所有测试条件下都远超此前基于LSTM的模型，且乱词测试排除了模型依赖语义选择偏好的可能性，说明模型确实学到了真正的句法规律。这一发现表明，纯注意力机制（self-attention）架构并非天然在句法敏感任务上处于劣势，反而可能比循环神经网络更擅长捕捉层级句法依存关系，为后续研究注意力机制如何编码句法结构提供了重要起点。
原文 arXiv:1901.05287;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.05287v1