OmniNet: Omnidirectional Representations from Transformers
Yi Tay Mostafa Dehghani Vamsi Aribandi Jai Gupta Philip Pham Zhen Qin Dara Bahri Da-Cheng Juan Donald Metzler
Abstract
This paper proposes Omnidirectional Representations from Transformers (OmniNet). In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network. This process can also be interpreted as a form of extreme or intensive attention mechanism that has the receptive field of the entire width and depth of the network. To this end, the omnidirectional attention is learned via a meta-learner, which is essentially another self-attention based model. In order to mitigate the computationally expensive costs of full receptive field attention, we leverage efficient self-attention models such as kernel-based (Choromanski et al., 2020), low-rank attention (Wang et al., 2020) and/or Big Bird (Zaheer et al., 2020) as the meta-learner. Extensive experiments are conducted on autoregressive language modeling (LM1B, C4), Machine Translation, Long Range Arena (LRA), and Image Recognition. The experiments show that OmniNet achieves considerable improvements across these tasks, including achieving state-of-the-art performance on LM1B, WMT’14 En-De/En-Fr, and Long Range Arena. Moreover, using omnidirectional representation in Visio
中文速览
Transformer在每一层只让词向量互相"看"同一层的其他词，而这篇论文想让每个词同时看到网络所有层、所有位置的表示，实现真正的全方位（omnidirectional）感受野。为此，作者引入了一个"元学习器"——本质上也是一个自注意力模块，把所有层的隐状态拼在一起统一处理，并借助核近似（Performer）、低秩投影（Linformer）、块稀疏（Big Bird）等高效注意力机制把原本爆炸式增长的计算量压回可接受范围。在语言模型（LM1B）、机器翻译（WMT'14英德/英法）、长序列理解基准（Long Range Arena）和图像识别（Vision Transformer）上的实验表明，OmniNet在多项任务上达到了当时最优水平，并且在少样本学习和微调场景下比原版ViT提升约3个百分点。这项工作的意义在于，它系统性地证明了"跨层全局注意力"是可以高效实现的，为后续设计更强的Transformer架构提供了一个新维度。
原文 arXiv:2103.01075;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.01075v1