GSNs: Generative Stochastic Networks
Guillaume Alain∗+, Yoshua Bengio∗+, Li Yao∗, Jason Yosinski†, Éric Thibodeau-Laufer∗, Saizheng Zhang∗ and Pascal Vincent∗ ∗ Department of Computer Science and Operations Research University of Montreal Montreal, H3C 3J7, Quebec, Canada † Department of Computer Science, Cornell University
Abstract
We introduce a novel training principle for generative probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework generalizes Denoising Auto-Encoders (DAE) and is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution is a conditional distribution that generally involves a small move, so it has fewer dominant modes and is unimodal in the limit of small moves. This simplifies the learning problem, making it less like density estimation and more akin to supervised function approximation, with gradients that can be obtained by backprop. The theorems provided here provide a probabilistic interpretation for denoising autoencoders and generalize them; seen in the context of this framework, auto-encoders that learn with injected noise are a special case of GSNs and can be interpreted as generative models. The theorems also provide an interesting justification for dependency networks and generalized pseudolikelihood and define an appropriate joint distribution and sampling mechanism, even when the conditionals are not consistent
中文速览
深度生成模型长期以来面临一个核心难题：训练时需要计算配分函数（partition function）或对大量模式（mode）求和，这在高维复杂分布下几乎无法处理。本文提出"生成随机网络"（Generative Stochastic Networks，GSN）框架，核心思路是放弃直接对数据分布 P(X) 建模，转而学习一条马尔可夫链（Markov chain）的转移算子——每一步转移只涉及局部小扰动，条件分布模式少、近似单峰，于是原本棘手的密度估计问题就变成了更易处理的监督式函数逼近，可以直接用反向传播（backprop）求梯度，彻底绕开不可计算的归一化常数。理论上，文章证明了只要转移分布估计一致，该马尔可夫链的平稳分布就是真实数据分布的一致估计，同时也为去噪自编码器（Denoising Auto-Encoder）提供了严格的生成模型解释，并将依赖网络（dependency network）纳入统一框架。实验表明，带隐变量的深层 GSN 在图像数据集上混合更快、生成质量更好，无需逐层预训练，意义在于为深度无监督生成模型的训练提供了一条可扩展、可用标准深度学习工具链直接优化的新路径。
原文 arXiv:1503.05571;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1503.05571v2