DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
Tao Shen† Jing Jiang†、Tianyi Zhou‡ Shirui Pan†、Guodong Long† Chengqi Zhang† \AND †Centre of Artificial Intelligence, FEIT, University of Technology Sydney ‡Paul G. Allen School of Computer Science、Engineering, University of Washington {guodong.long, jing.jiang, shirui.pan,
Abstract
Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, “Directional Self-Attention Network (DiSAN)”, is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by $1.02\%$ on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-ar
中文速览
自然语言处理领域长期依赖RNN捕捉长距离依赖、依赖CNN提取局部特征，但两者都存在计算慢或顺序信息建模受限的问题。为此，研究者提出了"方向性自注意力网络"（Directional Self-Attention Network，DiSAN），其核心创新在于设计了一种"多维方向性注意力"机制：注意力不再是每个词对的单一标量，而是逐特征计算的向量（多维），同时用位置掩码让注意力具有方向性（前向/后向），从而在不借助任何RNN或CNN结构的情况下同时捕捉上下文依赖和时序信息。整个DiSAN仅由方向性自注意力加上一个将序列压缩为向量的多维注意力模块构成，结构轻量、高度可并行。实验表明，DiSAN在SNLI自然语言推理数据集上超越所有句子编码方法，将最佳结果提升1.02%，并在SST情感分析、MultiNLI、SICK等多个基准上达到当时最优，同时参数更少、训练更快，证明纯注意力架构完全可以替代RNN/CNN成为通用句子编码骨干。
原文 arXiv:1709.04696;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.04696v3