Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
Junxian He, Daniel Spokoyny, Graham Neubig Carnegie Mellon University、Taylor Berg-Kirkpatrick University of California San Diego
Abstract
The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model’s posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods. In practice, however, VAE training often results in a degenerate local optimum known as “posterior collapse” where the model learns to ignore the latent variable and the approximate posterior mimics the prior. In this paper, we investigate posterior collapse from the perspective of training dynamics. We find that during the initial stages of training the inference network fails to approximate the model’s true posterior, which is a moving target. As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs. Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model’s current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update. Despit
中文速览
变分自编码器（VAE）在训练时常常陷入"后验崩塌"（posterior collapse）的困境——模型直接忽略潜变量、近似后验退化为先验，尤其在搭配LSTM等强自回归解码器处理文本时尤为严重。本文从训练动态的角度揭示了这一现象的根本原因：训练初期推断网络（inference network）严重滞后于生成模型的真实后验，这个"推断滞后"（inference lag）迫使生成模型为了与近似后验对齐而主动抛弃潜变量，最终锁死在崩塌的局部最优解中。基于此，作者提出了一个极其简单的改进方案——根据模型当前潜变量与观测之间的互信息，在每次更新生成模型之前对推断网络进行更激进的多步优化，无需修改模型结构或训练目标。实验表明，该方法在文本（Yahoo、Yelp）和图像（OMNIGLOT）基准上超越了强自回归基线，与更复杂的防崩塌方法效果相当，训练速度却快得多，为VAE训练提供了一个低成本、高效的实用解决方案。
原文 arXiv:1901.05534;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.05534v2