A Deep and Tractable Density Estimator
Benigno Uria Iain Murray School of Informatics, University of Edinburgh Hugo Larochelle Département d’informatique, Université de Sherbrooke
Abstract
The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.
中文速览
训练一个概率模型来精确回答"已知部分变量，求其他变量的概率"这类推断问题，一直很困难，因为传统NADE只固定一种变量排列顺序，换个推断任务就得用近似方法。本文提出"排序无关训练"（Order-Agnostic Training），通过共享参数并引入二值掩码（input mask），用随机梯度法同时优化所有可能排列顺序下的NADE模型，训练开销与单个模型相当。训练完成后，可以即时抽取最适合当前推断任务的排列顺序，还能随机抽取多个排列来组成集成模型（ensemble），进一步提升密度估计精度；更重要的是，该方案首次使深层NADE的训练计算量只随层数线性增长，而非此前令人望而却步的三次方复杂度。实验表明，深层NADE集成模型在多个基准数据集上达到当时最优的密度估计性能，为需要灵活推断能力的概率建模提供了实用工具。
原文 arXiv:1310.1757;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1310.1757v2