End-to-End Adversarial Text-to-Speech
Jeff Donahue, Sander Dieleman, Mikołaj Bińkowski, Erich Elsen, Karen Simonyan DeepMind Equal contribution. First author determined by coin toss.
Abstract
Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.111 Listen to our model reading this abstract at: htt
中文速览
语音合成长期依赖文本规范化、声学特征提取、声码器等多个独立模块串联完成，不仅流程繁琐还需要大量中间监督信号。这项工作提出了 EATS（端到端对抗文本转语音，End-to-end Adversarial Text-to-Speech），直接把字符或音素序列"一步到位"地转换成 24 kHz 原始音频波形。核心设计是一个可微的"对齐器"——通过预测每个输入词元的时长并累加得到单调对齐，再用高斯插值生成与音频同步的中间表示，从而在不需要预标注对齐信息的前提下解决输入输出长度严重不匹配的问题；训练时同时引入对抗损失、mel 频谱预测损失以及软性动态时间规整（soft DTW）来容忍语音时序的自然变化，确保模型既能生成真实感强的音频，又不被过度严格的帧级对齐约束所限制。最终模型在五分制主观评测（MOS）中达到 4.08 分，与依赖多阶段训练和额外标注的现有最优系统持平，同时推理速度可达实时的 200 倍，证明端到端对抗训练完全可以替代复杂的多模块 TTS 流水线。
原文 arXiv:2006.03575;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.03575v3