Transformers with convolutional context for ASR
Abstract
The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech “test clean” and “test other” subsets when no extra LM text is provided.111Code available at: github.com/pytorch/fairseq/ tree/master/examples/speech_recognition
中文速览
用卷积层替代传统正弦位置编码来给语音识别中的Transformer提供位置信息，是这项工作的核心出发点。研究者在编码器端使用二维卷积、在解码器端使用一维卷积，让模型从语音特征中自动学习局部上下文表示，再把这些带有隐式相对位置信息的特征输入Transformer层，由后者专注捕捉长程依赖关系。这种分工设计让训练更加稳定——无需学习率预热（warmup），用固定学习率1.0即可顺利收敛，极大简化了超参数调优。在1000小时Librispeech基准测试上，不借助任何外部语言模型文本的情况下，最终模型在"test clean"和"test other"两个子集上分别取得4.7%和12.9%的词错率，较此前同类条件下的最佳结果在困难子集上实现了12%至16%的相对提升，说明卷积位置编码是提升端到端语音识别系统稳健性与准确率的一条切实可行的路径。
原文 arXiv:1904.11660;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.11660v2