Star-Transformer
Qipeng Guo、Xipeng Qiu、Pengfei Liu \ANDYunfan Shao、Xiangyang Xue Shanghai Key Laboratory of Intelligent Information Processing, Fudan University School of Computer Science, Fudan University New York University、Zheng Zhang Corresponding Author. work done at NYU Shanghai, now with AWS Shanghai AI Lab.
Abstract
Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving the capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.
中文速览
标准Transformer虽然擅长捕捉长距离依赖关系，却因全连接的注意力机制导致计算复杂度随序列长度呈平方级增长，在中小规模数据集上还容易因参数过多而难以训练。为此，研究者提出了"星形Transformer"（Star-Transformer），将原本稠密的全连接拓扑替换为星形结构——所有词节点（卫星节点）围成一个环，并共享一个中继节点与每个词节点相连，从而让任意两个词之间的信息传递都只需经过"本地邻居→中继节点→目标词"两跳完成，连接数从O(n²)降至O(n)。在文本分类、自然语言推理和序列标注共22个数据集上的实验表明，Star-Transformer在中小规模数据集上持续优于标准Transformer，速度最高提升10倍，同时通过模拟任务验证了它与Transformer同样具备处理长距离依赖的能力。这项工作说明，把局部组合先验（环形连接）和全局信息汇聚（中继节点）合理分工，可以在不依赖大规模预训练的前提下，用更轻量的结构达到甚至超越标准Transformer的效果。
原文 arXiv:1902.09113;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.09113v3