Auxiliary Deep Generative Models
Lars Maaløe1 Casper Kaae Sønderby2 Søren Kaae Sønderby2 Ole Winther1,2 1Department of Applied Mathematics and Computer Science, Technical University of Denmark 2Bioinformatics Centre, Department of Biology, University of Copenhagen
Abstract
Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged but make the variational distribution more expressive. Inspired by the structure of the auxiliary variable we also propose a model with two stochastic layers and skip connections. Our findings suggest that more expressive and properly specified deep generative models converge faster with better results. We show state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.
中文速览
传统变分自编码器（VAE）因为用简单的对角高斯分布来近似后验，往往无法捕捉数据中复杂的依赖关系，导致模型表达能力受限、难以收敛。为此，研究者在深度生成模型中引入"辅助变量"（auxiliary variables），让推断网络的变分分布更灵活，同时保持原始生成模型不变；他们还在此基础上设计了带跳跃连接的两层随机隐变量模型（Skip Deep Generative Model, SDGM），使生成端也更具表达力。在玩具实验中，这些模型能拟合复杂的多峰后验分布；在半监督学习基准上，仅用极少量标注样本，就在 MNIST、SVHN 和 NORB 三个数据集上取得了当时最优的分类结果，且整个模型可端到端训练，无需预训练或人工特征工程。这项工作表明，让变分分布足够灵活、与真实后验匹配更好，是提升深度生成模型性能与训练稳定性的关键所在。
原文 arXiv:1602.05473;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1602.05473v4