Schrödinger’s Tree – On Syntax and Neural Language Models
Artur Kulmizev E-mail: Uppsala University Joakim Nivre Uppsala University
Abstract
In the last half-decade, the field of natural language processing (NLP) has undergone two major transitions: the switch to neural networks as the primary modeling paradigm and the homogenization of the training regime (pre-train, then fine-tune). Amidst this process, language models have emerged as NLP’s workhorse, displaying increasingly fluent generation capabilities and proving to be an indispensable means of knowledge transfer downstream. Due to the otherwise opaque, black-box nature of such models, researchers have employed aspects of linguistic theory in order to characterize their behavior. Questions central to syntax — the study of the hierarchical structure of language — have factored heavily into such work, shedding invaluable insights about models’ inherent biases and their ability to make human-like generalizations. In this paper, we attempt to take stock of this growing body of literature. In doing so, we observe a lack of clarity across numerous dimensions, which influences the hypotheses that researchers form, as well as the conclusions they draw from their findings. To remedy this, we urge researchers make careful considerations when investigating coding properties,
中文速览
过去几年里，预训练语言模型（如BERT、GPT）横扫自然语言处理（NLP）各项任务，但这些"黑箱"模型到底学没学会语言的句法结构（syntax），学界至今众说纷纭、结论相互矛盾。这篇综述系统梳理了三大主流评测范式——目标句法评估（targeted syntactic evaluation）、探针分析（probing）和下游任务评估——并指出现有研究普遍存在概念模糊的问题：研究者往往混淆了抽象的句法结构与词序、形态屈折等具体的编码属性（coding properties），忽视了不同语言在句法实现方式上的巨大差异，还过度依赖聚合指标（aggregate metrics）把复杂的句法现象简单化。作者呼吁研究者在选择编码属性、表征形式和评估任务时保持更清晰的概念区分，并对不同类型研究问题的适用边界加以说明。这项工作的价值在于为整个领域提供了一套更精细的分析框架，有助于消解现有文献中的表面矛盾，推动学界对语言模型句法能力形成更有层次、更少以偏概全的认识。
原文 arXiv:2110.08887;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.08887v1