ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech
Rongjie Huang∗, Zhou Zhao†, Huadai Liu∗, Jinglin Liu, Chenye Cui, Yi Ren Zhejiang University
Abstract
Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hinder their applications to text-to-speech deployment. Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling. In this work, we propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. Unlike previous work estimating the gradient for data density, ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. To tackle the model convergence challenge with decreased diffusion iterations, ProDiff reduces the data variance in the target site via knowledge distillation. Specifically, the denoising model uses the generated mel-spectrogram from an N-step DDIM teacher as the training target and distills the behavior into a new model with N/2 steps. As such, it allows the TTS model to make sharp predictions and further reduces t
中文速览
文本转语音（TTS）领域的扩散概率模型虽然生成质量出色，但反复迭代去噪的过程极其耗时，难以实际部署。ProDiff 针对这一痛点，放弃了主流的"梯度预测"参数化方式，改为让去噪网络直接预测干净的梅尔频谱（mel-spectrogram），从而在大幅缩减迭代步数时避免明显的质量下降；同时借助知识蒸馏（knowledge distillation），用 N 步 DDIM 教师模型生成的低方差输出作为训练目标，将模型压缩到只需 N/2 步，如此递进直至仅需 2 次迭代即可完成高保真合成。实验表明，ProDiff 在单张 NVIDIA 2080Ti 上实现了比实时快 24 倍的推理速度，同时样本质量和多样性与需要数百步的顶尖模型持平，是首个让扩散模型真正满足实时 TTS 部署需求的方案。
原文 arXiv:2207.06389;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.06389v1