STATE-OF-THE-ART SPEECH RECOGNITION USING MULTI-STREAM SELF-ATTENTION WITH DILATED 1D CONVOLUTIONS
Abstract
Self-attention has been a huge success for many downstream tasks in NLP, which led to exploration of applying self-attention to speech problems as well. The efficacy of self-attention in speech applications, however, seems not fully blown yet since it is challenging to handle highly correlated speech frames in the context of self-attention. In this paper we propose a new neural network model architecture, namely multi-stream self-attention, to address the issue thus make the self-attention mechanism more effective for speech recognition. The proposed model architecture consists of parallel streams of self-attention encoders, and each stream has layers of 1D convolutions with dilated kernels whose dilation rates are unique given stream, followed by a self-attention layer. The self-attention mechanism in each stream pays attention to only one resolution of input speech frames and the attentive computation can be more efficient. In a later stage, outputs from all the streams are concatenated then linearly projected to the final embedding. By stacking the proposed multi-stream self-attention encoder blocks and rescoring the resultant lattices with neural network language models, we ach
中文速览
语音识别中把自注意力（self-attention）直接用于连续语音帧效果有限，根本原因在于相邻语音帧高度相关、缺乏像词语那样清晰的语义边界。为解决这一问题，研究者提出了"多流自注意力"（multi-stream self-attention）架构：将多个并行的编码器流同时运行，每条流用不同膨胀率（dilation rate）的一维卷积在不同分辨率上提取局部语音特征，再送入各自的多头自注意力层，最后将所有流的输出拼接并线性投影为最终表示，从而让每条流只专注于一种时间分辨率的模式。在LibriSpeech标准测试集上，结合神经网络语言模型重打分，该系统在test-clean上取得2.2%的词错误率（WER），刷新了当时的最优记录。这项工作不仅为语音识别提供了一种更高效的自注意力建模方式，也说明多分辨率并行处理对捕捉不同类型音素的时序边界至关重要。
原文 arXiv:1910.00716;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.00716v1