Tree Transformer: Integrating Tree Structures into Self-Attention
Yau-Shian Wang Hung-Yi Lee Yun-Nung Chen National Taiwan University, Taipei, Taiwan
Abstract
Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks. However, it is unclear what the learned attention captures. The attention computed by attention heads seems not to match human intuitions about hierarchical structures. This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures. The tree structures can be automatically induced from raw texts by our proposed “Constituent Attention” module, which is simply implemented by self-attention between two adjacent words. With the same training procedure identical to BERT, the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures, better language modeling, and further learning more explainable attention scores111The source code is publicly available at https://github.com/yaushian/Tree-Transformer..
中文速览
语言有天然的层次结构（比如词组、句子成分），但主流的Transformer注意力机制完全忽视了这一点，导致注意力矩阵难以解释、也不符合人类对句子结构的直觉。为此，研究者提出了Tree Transformer：在双向Transformer编码器的每一层插入一个"成分注意力"（Constituent Attention）模块，用相邻词之间的自注意力自动从原始文本中归纳出成分树结构，再以此为约束让注意力头只在同一成分内部聚焦，并随层数加深将小成分逐步合并为大成分。在与BERT完全相同的无监督预训练流程下，Tree Transformer在无监督句法分析、掩码语言模型困惑度以及注意力可解释性三个维度上均优于原始Transformer。这项工作的意义在于：无需任何人工标注的句法树，就能让大规模预训练模型同时学到更好的语言表示和更贴近人类语言直觉的层次结构，为理解"Transformer究竟学到了什么"提供了新思路。
原文 arXiv:1909.06639;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.06639v2