Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
Abstract
In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks based on self-attention are used to encode both audio and label sequences independently. The activations from both audio and label encoders are combined with a feed-forward layer to compute a probability distribution over the label space for every combination of acoustic frame position and label history. This is similar to the Recurrent Neural Network Transducer (RNN-T) model, which uses RNNs for information encoding instead of Transformer encoders. The model is trained with the RNN-T loss well-suited to streaming decoding. We present results on the LibriSpeech dataset showing that limiting the left context for self-attention in the Transformer layers makes decoding computationally tractable for streaming, with only a slight degradation in accuracy. We also show that the full attention version of our model beats the-state-of-the art accuracy on the LibriSpeech benchmarks. Our results also show that we can bridge the gap between full attention and limited attention versions of our model by attending to a l
中文速览
流式语音识别（streaming ASR）长期依赖循环神经网络（RNN），但RNN训练慢、并行性差；而Transformer虽然效果好，却因自注意力需要看到全部输入，天然不适合实时流式场景。本文提出"Transformer转录器（Transformer Transducer，T-T）"，把传统RNN-T架构中的LSTM编码器替换为Transformer编码器，同时保留RNN-T的对齐损失函数，让音频编码器和标签编码器各自独立做自注意力，而不是跨两者做交叉注意力，从而避免了流式解码的障碍；为了让每帧计算量恒定，进一步将注意力限制在固定长度的历史窗口内，并引入相对位置编码以便复用已有状态。在LibriSpeech基准上，全注意力版T-T刷新了当时的最优词错率，限窗版性能仅略有下降，而且训练速度比同参数量的LSTM-RNN-T快约3.5倍，证明Transformer可以真正落地于低延迟的端到端流式语音识别系统。
原文 arXiv:2002.02562;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.02562v2