Accelerating Diffusion Models via Early Stop of the Diffusion Process
Zhaoyang Lyu1,2 Xudong Xu1,2 Ceyuan Yang1,2 Dahua Lin1,2 Bo Dai2 1The Chinese University of Hong Kong 2Shanghai AI Laboratory
Abstract
Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive performance on various generation tasks. By modeling the reverse process of gradually diffusing the data distribution into a Gaussian distribution, generating a sample in DDPMs can be regarded as iteratively denoising a randomly sampled Gaussian noise. However, in practice DDPMs often need hundreds or even thousands of denoising steps to obtain a high-quality sample from the Gaussian noise, leading to extremely low inference efficiency. In this work, we propose a principled acceleration strategy, referred to as Early-Stopped DDPM (ES-DDPM), for DDPMs. The key idea is to stop the diffusion process early where only the few initial diffusing steps are considered and the reverse denoising process starts from a non-Gaussian distribution. By further adopting a powerful pre-trained generative model, such as GAN and VAE, in ES-DDPM, sampling from the target non-Gaussian distribution can be efficiently achieved by diffusing samples obtained from the pre-trained generative model. In this way, the number of required denoising steps is significantly reduced. In the meantime, the sample quality of ES-DDPM also improves su
中文速览
扩散模型（Denoising Diffusion Probabilistic Models, DDPM）生成质量出色，但每次采样需要上千步去噪迭代，推理速度极慢。本文提出"早停扩散模型"（Early-Stopped DDPM, ES-DDPM）：不再把数据一路扩散成纯高斯噪声，而是只走前几步扩散就停下，让去噪过程从一个接近真实数据的非高斯分布出发，从而大幅缩短去噪步数；为了能高效地从这个非高斯分布采样，ES-DDPM借助预训练的生成模型（GAN或VAE）生成粗略图像，再加上少量扩散噪声来近似该分布，整个过程几乎不引入额外计算开销。在CIFAR-10、CelebA、ImageNet等多个基准上，ES-DDPM不仅将去噪步数从1000步大幅压缩至约100步，生成质量还同时超越了原始DDPM和所借助的预训练模型，且该方法可与其他加速策略叠加使用，并额外支持对生成图像的全局语义与局部像素级双重控制，为扩散模型的实用化提供了原则性的解决思路。
原文 arXiv:2205.12524;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.12524v2