Importance Weighted Autoencoders
Yuri Burda, Roger Grosse、Ruslan Salakhutdinov Department of Computer Science University of Toronto Toronto, ON, Canada
Abstract
The variational autoencoder (VAE; Kingma & Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network’s entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.
中文速览
变分自编码器（VAE）在生成建模中表现出色，但它对后验分布做了较强假设（近似因式分解且可用神经网络预测），导致模型容量未被充分利用、学到的潜在表示过于简单。为此，作者提出了重要性加权自编码器（Importance Weighted Autoencoder, IWAE），它与VAE架构完全相同，区别仅在于训练目标：用基于重要性采样的多样本对数似然下界替代VAE的单样本变分下界，样本数越多下界越紧、越接近真实对数似然。识别网络（recognition network）同时生成多个后验近似样本，通过重要性权重加权平均来更灵活地逼近复杂后验，从而放宽了VAE对后验形式的限制。实验表明，IWAE在密度估计基准上取得了显著更高的测试对数似然，并学到了更丰富的潜在空间表示，证明这一改动简单却有效，为后续改进深度生成模型的训练目标提供了重要参考。
原文 arXiv:1509.00519;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1509.00519v4