Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
Rewon Child OpenAI San Francisco, CA
Abstract
We present a hierarchical VAE that, for the first time, generates samples quickly and outperforms the PixelCNN in log-likelihood on all natural image benchmarks. We begin by observing that, in theory, VAEs can actually represent autoregressive models, as well as faster, better models if they exist, when made sufficiently deep. Despite this, autoregressive models have historically outperformed VAEs in log-likelihood. We test if insufficient depth explains why by scaling a VAE to greater stochastic depth than previously explored and evaluating it CIFAR-10, ImageNet, and FFHQ. In comparison to the PixelCNN, these very deep VAEs achieve higher likelihoods, use fewer parameters, generate samples thousands of times faster, and are more easily applied to high-resolution images. Qualitative studies suggest this is because the VAE learns efficient hierarchical visual representations. We release our source code and models at https://github.com/openai/vdvae.
中文速览
长期以来，自回归模型（autoregressive model）如 PixelCNN 在图像生成的对数似然（log-likelihood）评估上一直优于变分自编码器（VAE, Variational Autoencoder），但自回归模型生成速度极慢、难以扩展到高分辨率图像。研究者从理论上证明，只要堆叠足够多的随机层（stochastic layers），层次化 VAE 其实完全可以表达自回归模型的建模能力，过去 VAE 表现不佳的根本原因在于深度不够。为此，他们设计了一套专门支持超深层训练的架构——通过残差缩放、最近邻上采样和梯度跳过等技巧将 VAE 扩展到最多 78 个随机层，而此前文献最多只探索过 30 层。在 CIFAR-10、ImageNet 和 FFHQ 等主流自然图像基准上，这个"极深 VAE"（VDVAE）首次在对数似然上全面超越 PixelCNN，同时参数量更少、采样速度快数千倍，还能轻松处理高分辨率图像。这一结果说明，更深的层次化潜变量结构能让模型自发学到高效的视觉层级表示，为以后用更小、更快的生成模型替代昂贵的自回归大模型提供了有力依据。
原文 arXiv:2011.10650;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.10650v2