Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Tal Linzen1,2 Emmanuel Dupoux1 LSCP1、IJN2, CNRS, EHESS and ENS, PSL Research University {tal.linzen, \AndYoav Goldberg Computer Science Department Bar Ilan University
Abstract
The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. Linguistic regularities are often sensitive to syntactic structure; can such dependencies be captured by LSTMs, which do not have explicit structural representations? We begin addressing this question using number agreement in English subject-verb dependencies. We probe the architecture’s grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when sequential and structural information conflicted. The frequency of such errors rose sharply in the language-modeling setting. We conclude that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with m
中文速览
长短期记忆网络（LSTM）虽然在语言处理任务上表现出色，但它本身并不包含任何句法结构的显式表示，研究者想知道它究竟能在多大程度上学会依赖句法结构的语法规则。为此，本文以英语主谓数一致（subject-verb number agreement）为切入点，分别在"直接预测动词数"、"可接受性判断"和"语言模型"三种监督强度递减的设置下，系统测试了LSTM对这一结构敏感依存关系的掌握情况。结果显示，在最强监督（直接预测动词单复数）下，LSTM总体错误率不足1%，但一旦出现"吸引词"（与主语数不一致的介入名词），错误率就会显著上升；切换到无句法标注的语言模型目标后，LSTM在困难样例上的表现甚至低于随机水平，错误数是强监督模型的五倍以上。这一发现表明，仅靠语言建模信号不足以让LSTM捕捉语法结构依存关系，若下游任务（如机器翻译、文本生成）需要句法敏感能力，就必须引入更直接的监督信号来加以补充。
原文 arXiv:1611.01368;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.01368v1