Self-Attentional Acoustic Models
Abstract
Self-attention is a method of encoding sequences of vectors by relating these vectors to each-other based on pairwise similarities. These models have recently shown promising results for modeling discrete sequences, but they are non-trivial to apply to acoustic modeling due to computational and modeling issues. In this paper, we apply self-attention to acoustic modeling, proposing several improvements to mitigate these issues: First, self-attention memory grows quadratically in the sequence length, which we address through a downsampling technique. Second, we find that previous approaches to incorporate position information into the model are unsuitable and explore other representations and hybrid models to this end. Third, to stress the importance of local context in the acoustic signal, we propose a Gaussian biasing approach that allows explicit control over the context range. Experiments find that our model approaches a strong baseline based on LSTMs with network-in-network connections while being much faster to compute. Besides speed, we find that interpretability is a strength of self-attentional acoustic models, and demonstrate that self-attention heads learn a linguistically
中文速览
用自注意力（self-attention）机制来替代传统循环神经网络（RNN）做语音识别的声学建模，面临序列过长导致显存爆炸、位置信息难以融入、以及局部上下文建模不足三大挑战。针对这三个问题，研究者分别提出了序列重塑降采样方案、混合自注意力与LSTM的架构设计，以及带可学习方差的高斯偏置注意力机制。在200小时TED演讲语料库上的实验表明，所提模型在词错误率上接近强基线LSTM+网络内网络模型，训练速度却快得多，高斯偏置方案最多将词错误率降低1.59%。更值得关注的是，不同注意力头自发学会了按音素类别分工协作的可解释语言学模式，为理解端到端语音模型的内部机制提供了难得的窗口。
原文 arXiv:1803.09519;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.09519v2