RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
Abstract
In recent years, all-neural end-to-end approaches have obtained state-of-the-art results on several challenging automatic speech recognition (ASR) tasks. However, most existing works focus on building ASR models where train and test data are drawn from the same domain. This results in poor generalization characteristics on mismatched-domains: e.g., end-to-end models trained on short segments perform poorly when evaluated on longer utterances. In this work, we analyze the generalization properties of streaming and non-streaming recurrent neural network transducer (RNN-T) based end-to-end models in order to identify model components that negatively affect generalization performance. We propose two solutions: combining multiple regularization techniques during training, and using dynamic overlapping inference. On a long-form YouTube test set, when the non-streaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from $22.3\%$ to $14.8\%$ ; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from $67.0\%$ to $25.3\%$ . Finally, when trained on Librispeech, we
中文速览
针对长音频场景，端到端语音识别（ASR）模型在训练数据都是短片段时会严重过拟合，导致在YouTube长视频等场景下出现大量漏词、词错率飙升。研究者通过逐步冻结模型各组件的实验，锁定了"编码器（encoder）过拟合"是根本症结所在——编码器在面对比训练时更长的音频时，对空白符（blank token）的预测变得极不稳定，进而引发集束搜索（beam search）崩溃。为此，论文提出两类解决方案：一是在训练阶段将多种正则化技术（数据增强、Dropout、多条件训练等）组合使用，称为"正则化鸡尾酒"；二是在推理阶段采用动态重叠滑窗（dynamic overlapping inference），将长音频切成带重叠的短段分别识别后再拼合。实验结果表明，用短片段YouTube数据训练的非流式RNN-T模型词错率从22.3%降至14.8%，用搜索查询数据训练的流式模型在YouTube测试集上从67.0%降至25.3%，而仅在LibriSpeech上训练的模型在YouTube测试集词错率更从99.8%大幅压缩到33.0%，充分说明这套方法对跨域长音频泛化具有显著且普适的价值。
原文 arXiv:2005.03271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.03271v3