End-to-End ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Gabriel Synnaeve Facebook, NYC、Qiantong Xu∗ Facebook, Menlo Park、Jacob Kahn∗ Facebook, Menlo Park、Tatiana Likhomanenko∗ Facebook, Menlo Park、Edouard Grave∗ Facebook, Paris、Vineel Pratap Facebook, Menlo Park、Anuroop Sriram Facebook, Menlo Park、Vitaliy Liptchinsky Facebook, Menlo Park、Ronan Collobert∗ Facebook, Menlo Park Equal contribution.
Abstract
We study pseudo-labeling for the semi-supervised training of ResNet, Time-Depth Separable ConvNets, and Transformers for speech recognition, with either CTC or Seq2Seq loss functions. We perform experiments on the standard LibriSpeech dataset, and leverage additional unlabeled data from LibriVox through pseudo-labeling. We show that while Transformer-based acoustic models have superior performance with the supervised dataset alone, semi-supervision improves all models across architectures and loss functions and bridges much of the performance gaps between them. In doing so, we reach a new state-of-the-art for end-to-end acoustic models decoded with an external language model in the standard supervised learning setting, and a new absolute state-of-the-art with semi-supervised training. Finally, we study the effect of leveraging different amounts of unlabeled audio, propose several ways of evaluating the characteristics of unlabeled audio which improve acoustic modeling, and show that acoustic models trained with more audio rely less on external language models.
中文速览
语音识别领域的端到端模型（end-to-end ASR）虽然结构简单，但识别准确率长期落后于依赖强制对齐的传统流程，研究者希望弄清楚能否在不改变算法核心的前提下让端到端模型追上传统系统。本文系统评估了伪标签（pseudo-labeling）半监督训练方法：先用有标注的 LibriSpeech 数据训练声学模型，再用该模型对 LibriVox 中约 5.4 万小时无标注音频生成伪标签，然后将这些伪标签数据一并用于重新训练 ResNet、时深可分离卷积（TDS ConvNet）和 Transformer 三类模型，并同时考察 CTC 和 Seq2Seq 两种损失函数。实验结果表明，Transformer 在纯监督场景下表现最佳，但引入半监督训练后，所有架构和损失函数的性能都大幅提升，彼此间的差距也随之显著缩小，最终在 LibriSpeech test-clean / test-other 上分别达到 2.09% / 4.11% 的词错率（WER），刷新了端到端模型的最佳纪录。这项工作的意义在于，它表明大规模伪标签半监督训练本身就足以让结构简单的端到端模型与复杂传统系统媲美，为未来通过数据扩充而非复杂架构搜索来提升语音识别性能提供了有力支撑。
原文 arXiv:1911.08460;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.08460v3