FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis
Rongjie Huang1111Work done during internship at Tencent AI Lab222Equal contribution Max W. Y. Lam Jun Wang2 Dan Su2 Dong Yu3 Yi Ren1 Zhou Zhao1333Corresponding author 1Zhejiang University 2Tencent AI Lab, China 3Tencent AI Lab, USA {rongjiehuang, rayeren, {maxwylam, joinerwang, dansu,
Abstract
Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hindered their applications to speech synthesis. This paper proposes FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employs a stack of time-aware location-variable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. A noise schedule predictor is also adopted to reduce the sampling steps without sacrificing the generation quality. Based on FastDiff, we design an end-to-end text-to-speech synthesizer, FastDiff-TTS, which generates high-fidelity speech waveforms without any intermediate feature (e.g., Mel-spectrogram). Our evaluation of FastDiff demonstrates the state-of-the-art results with higher-quality (MOS 4.28) speech samples. Also, FastDiff enables a sampling speed of 58x faster than real-time on a V100 GPU, making diffusion models practically applicable to speech synthesis deployment for the first time. We further show that FastDiff generalized well to the mel-spectrogram inversion of unseen s
中文速览
扩散模型生成语音质量高但速度极慢，FastDiff 就是为了打破这一瓶颈而提出的。它的核心做法有两个：一是设计了一种"时间感知位置可变卷积"（time-aware location-variable convolution），让模型在去噪时能同时感知扩散步骤和声学特征，从而用多样化的感受野捕捉语音的长程依赖；二是引入噪声调度预测器，把原本需要数百乃至数千步的采样过程压缩到仅 4 步，同时保持生成质量。在实验中，FastDiff 在平均主观评分（MOS）上以 4.28 分超过了 WaveNet 等强基线，并在 V100 GPU 上实现了比实时快 58 倍的合成速度，首次让扩散模型真正满足实际语音合成部署的需求；在此基础上构建的端到端系统 FastDiff-TTS 更可直接从音素生成波形，无需梅尔频谱图等中间特征，进一步简化了语音合成流程。
原文 arXiv:2204.09934;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.09934v1