Bilateral Denoising Diffusion Models
Max W. Y. Lam, Jun Wang, Rongjie Huang, Dan Su Tencent AI Lab Shenzhen, China {maxwylam,、Dong Yu Tencent AI Lab Bellevue WA, USA Work done during an internship at Tencent AI Lab.
Abstract
Denoising diffusion probabilistic models (DDPMs) have emerged as competitive generative models yet brought challenges to efficient sampling. In this paper, we propose novel bilateral denoising diffusion models (BDDMs), which take significantly fewer steps to generate high-quality samples. From a bilateral modeling objective, BDDMs parameterize the forward and reverse processes with a score network and a scheduling network, respectively. We show that a new lower bound tighter than the standard evidence lower bound can be derived as a surrogate objective for training the two networks. In particular, BDDMs are efficient, simple-to-train, and capable of further improving any pre-trained DDPM by optimizing the inference noise schedules. Our experiments demonstrated that BDDMs can generate high-fidelity samples with as few as 3 sampling steps and produce comparable or even higher quality samples than DDPMs using 1000 steps with only 16 sampling steps (a 62x speedup).
中文速览
扩散概率模型（DDPM）虽然能生成高质量样本，但每次采样需要走上千步、速度极慢。为此，研究者提出了双边去噪扩散模型（Bilateral Denoising Diffusion Models，BDDM），核心思路是把原本固定不变的噪声调度表（noise schedule）也交给一个专门的"调度网络"来学习，使其在推理阶段自动找到最优的、更短的噪声序列，同时保留原有的"评分网络"负责去噪，两者联合训练。理论上，作者推导出一个比标准证据下界（ELBO）更紧的新下界作为训练目标，并证明DDPM原有的训练启发式方法正是该目标的一个特例，从而为其提供了理论依据。实验结果表明，BDDM只需3步就能生成高保真语音样本，仅用16步便能达到甚至超过原始DDPM 1000步的生成质量（加速约62倍），且该方法可直接插入任意已训练好的DDPM模型，几乎不增加额外训练开销。这项工作从根本上解决了扩散模型推理慢的瓶颈，为其在实时语音合成等对速度敏感的场景中落地铺平了道路。
原文 arXiv:2108.11514;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2108.11514v3