Iterative Neural Autoregressive Distribution Estimator (NADE-k𝑘k)
Tapani Raiko Aalto University、Li Yao Université de Montréal、KyungHyun Cho Université de Montréal、Yoshua Bengio Université de Montréal, CIFAR Senior Fellow
Abstract
Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in $k$ steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-prediction training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.
中文速览
神经自回归密度估计器（NADE）在重建缺失数据时只做一步推断，而一步到位的重建往往不够精准。本文提出的 NADE-k 将这一推断过程扩展为 k 步迭代：每一步利用同一组共享权重的神经网络，在上一步的重建结果基础上再做一次改进，相当于对隐变量模型执行 k 轮变分推断。这样既保留了 NADE 可精确计算似然、能直接采样的优良特性，又借鉴了多预测训练的思路，让模型"学会逐步改进"而非"一次猜对"。在二值化 MNIST 手写数字和 Caltech 101 剪影两个基准数据集上，NADE-k（k=5）的密度估计性能超越了原始 NADE 及其带掩码改进版，达到与深度置信网络相当的水平，为构建既理论清晰又实用高效的无监督深度学习模块提供了新思路。
原文 arXiv:1406.1485;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1406.1485v3