High-Dimensional Probability Estimation with Deep Density Models
Oren Rippel111http://math.mit.edu/r̃ippel Ryan Prescott Adams222http://people.seas.harvard.edu/r̃pa Massachusetts Institute of Technology, Harvard University Harvard University
Abstract
One of the fundamental problems in machine learning is the estimation of a probability distribution from data. Many techniques have been proposed to study the structure of data, most often building around the assumption that observations lie on a lower-dimensional manifold of high probability. It has been more difficult, however, to exploit this insight to build explicit, tractable density models for high-dimensional data. In this paper, we introduce the deep density model (DDM), a new approach to density estimation. We exploit insights from deep learning to construct a bijective map to a representation space, under which the transformation of the distribution of the data is approximately factorized and has identical and known marginal densities. The simplicity of the latent distribution under the model allows us to feasibly explore it, and the invertibility of the map to characterize contraction of measure across it. This enables us to compute normalized densities for out-of-sample data. This combination of tractability and flexibility allows us to tackle a variety of probabilistic tasks on high-dimensional datasets, including: rapid computation of normalized densities at test-tim
中文速览
用深度神经网络构造一个可逆映射（双射），把高维复杂数据"变换"到一个结构简单、各维度近似独立的隐空间，这就是本文提出的深度密度模型（Deep Density Model, DDM）的核心思路。由于映射是严格可逆的，可以通过计算雅可比行列式把隐空间中的简单概率密度"折算"回原始数据空间，从而得到无需估计配分函数的归一化概率密度。实验表明，该模型能在 MNIST、CIFAR 等高维数据集上直接计算测试样本的密度、无需 MCMC 地生成新样本，并可用于估计数据的联合熵及构建贝叶斯分类器。这一工作的价值在于，它把流形学习与概率密度估计统一在同一框架下，兼顾了模型的灵活性与概率的可解释性，为高维数据的生成建模和不确定性量化提供了一条切实可行的路径。
原文 arXiv:1302.5125;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1302.5125v1