DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs
Songxiang Liu Dan Su Dong Yu
Abstract
Denoising diffusion probabilistic models (DDPMs) are expressive generative models that have been used to solve a variety of speech synthesis problems. However, because of their high sampling costs, DDPMs are difficult to use in real-time speech processing applications. In this paper, we introduce DiffGAN-TTS, a novel DDPM-based text-to-speech (TTS) model achieving high-fidelity and efficient speech synthesis. DiffGAN-TTS is based on denoising diffusion generative adversarial networks (GANs), which adopt an adversarially-trained expressive model to approximate the denoising distribution. We show with multi-speaker TTS experiments that DiffGAN-TTS can generate high-fidelity speech samples within only 4 denoising steps. We present an active shallow diffusion mechanism to further speed up inference. A two-stage training scheme is proposed, with a basic TTS acoustic model trained at stage one providing valuable prior information for a DDPM trained at stage two. Our experiments show that DiffGAN-TTS can achieve high synthesis performance with only 1 denoising step.
中文速览
语音合成领域长期面临一个两难困境：扩散概率模型（DDPM）虽然能高质量地拟合复杂的"一对多"文本到声学特征映射，却因为需要成百上千次去噪迭代而无法用于实时场景。DiffGAN-TTS 的解决思路是用一个对抗训练的声学生成器（GAN）来替代传统 DDPM 中的高斯去噪分布假设，从而允许每步去噪跨越更大的幅度，把迭代次数压缩到仅 4 步甚至 1 步；在此基础上，论文还提出"主动浅扩散（active shallow diffusion）"机制：先用第一阶段训练的基础声学模型生成一个粗糙但有价值的先验梅尔频谱，再让第二阶段的扩散模型从这个先验出发而非纯噪声出发做反向去噪，进一步缩短推理路径。实验在多说话人合成任务上表明，DiffGAN-TTS 仅用 1 步去噪即可达到与多步基线相当甚至更优的自然度和保真度。这项工作的意义在于，它首次将扩散模型的高表现力与 GAN 的高效采样结合进多说话人 TTS 框架，为扩散模型在实时语音处理中的落地提供了切实可行的路径。
原文 arXiv:2201.11972;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.11972v1