GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesis
Abstract
Recent advances in neural multi-speaker text-to-speech (TTS) models have enabled the generation of reasonably good speech quality with a single model and made it possible to synthesize the speech of a speaker with limited training data. Fine-tuning to the target speaker data with the multi-speaker model can achieve better quality, however, there still exists a gap compared to the real speech sample and the model depends on the speaker. In this work, we propose GANSpeech, which is a high-fidelity multi-speaker TTS model that adopts the adversarial training method to a non-autoregressive multi-speaker TTS model. In addition, we propose simple but efficient automatic scaling methods for feature matching loss used in adversarial training. In the subjective listening tests, GANSpeech significantly outperformed the baseline multi-speaker FastSpeech and FastSpeech2 models, and showed a better MOS score than the speaker-specific fine-tuned FastSpeech2.
中文速览
多说话人语音合成（multi-speaker TTS）模型在训练数据有限时往往生成质量不足，即便针对目标说话人做微调也仍与真实语音有差距。GANSpeech 在非自回归多说话人 TTS 模型的基础上引入对抗训练（GAN），以带说话人条件的联合条件/无条件判别器（JCU discriminator）来学习梅尔频谱图的全局特征和说话人特有特征，同时提出"自动缩放特征匹配损失"（scaled feature matching loss），在训练中动态平衡特征匹配损失与重建损失，从而避免训练不稳定和模式崩溃。主观听感测试（MOS）显示，GANSpeech 显著优于基线多说话人 FastSpeech 和 FastSpeech2，甚至超越了针对特定说话人精调的 FastSpeech2，且无需任何说话人专属微调或外部模型。这项工作表明，将对抗训练与自动损失缩放相结合，能在不增加系统复杂度的情况下大幅提升多说话人语音合成的自然度，对少样本和零样本说话人适配场景具有重要参考价值。
原文 arXiv:2106.15153;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.15153v1