Generative Modeling by Estimating Gradients of the Data Distribution
Yang Song Stanford University、Stefano Ermon Stanford University
Abstract
We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.
中文速览
用多尺度噪声扰动数据、再用退火朗之万动力学（annealed Langevin dynamics）采样，是这项工作解决生成模型难题的核心思路。现有的基于得分匹配（score matching）的生成方法在真实数据上有两大硬伤：当数据集中在低维流形上时，得分函数在环境空间里根本没有定义；而在低密度区域，训练样本稀少导致得分估计严重失准，朗之万动力学也很难在不同模式间正确混合。为此，作者提出用一系列从大到小的高斯噪声对数据进行扰动，训练一个以噪声等级为条件的统一得分网络，再在采样时像模拟退火一样从高噪声水平逐步降低噪声、引导样本落向真实数据流形。整套方法无需对抗训练、无需在训练中做MCMC采样，架构也高度灵活，在CIFAR-10上以无条件生成的初始得分（inception score）8.87刷新了当时的最优记录，同时在图像修复任务上也展现出强大的表示学习能力，为后来扩散模型（diffusion models）的兴起奠定了重要基础。
原文 arXiv:1907.05600;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.05600v3