A Continuous Time Framework for Discrete Denoising Models
Andrew Campbell1、Joe Benton1、Valentin De Bortoli2 Tom Rainforth1、George Deligiannidis1、Arnaud Doucet1
Abstract
We provide the first complete continuous time framework for denoising diffusion models of discrete data. This is achieved by formulating the forward noising process and corresponding reverse time generative process as Continuous Time Markov Chains (CTMCs). The model can be efficiently trained using a continuous time version of the ELBO. We simulate the high dimensional CTMC using techniques developed in chemical physics and exploit our continuous time framework to derive high performance samplers that we show can outperform discrete time methods for discrete data. The continuous time treatment also enables us to derive a novel theoretical result bounding the error between the generated sample distribution and the true data distribution.
中文速览
离散数据（文本、图像像素值、音乐音符等）的扩散生成模型此前只能在离散时间步下训练和采样，这限制了采样灵活性、理论分析能力和生成质量。本文将离散数据的去噪扩散过程完整地建立在连续时间框架下：把前向加噪和反向生成都表示为连续时间马尔可夫链（Continuous Time Markov Chain, CTMC），推导出对应的连续时间证据下界（ELBO）用于高效训练，并借鉴化学物理中的tau-leaping技术设计了一种新型预测-校正采样器来高效模拟高维反向过程。理论上，该框架还首次给出了离散状态生成分布与真实数据分布之间误差的定量上界。在CIFAR-10图像和单声部音乐序列的实验中，新采样器的生成质量超越了此前所有离散时间离散状态方法，大幅缩小了离散数据建模与连续数据建模之间的性能差距，为离散生成模型提供了一套理论完备、实践高效的新基础。
原文 arXiv:2205.14987;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.14987v2