A RAD approach to deep mixture models
Laurent Dinh \addrGoogle Brain \ANDJascha Sohl-Dickstein \addrGoogle Brain \ANDHugo Larochelle \addrGoogle Brain \ANDRazvan Pascanu \addrDeepMind
Abstract
Flow based models such as Real NVP are an extremely powerful approach to density estimation. However, existing flow based models are restricted to transforming continuous densities over a continuous input space into similarly continuous distributions over continuous latent variables. This makes them poorly suited for modeling and representing discrete structures in data distributions, for example class membership or discrete symmetries. To address this difficulty, we present a normalizing flow architecture which relies on domain partitioning using locally invertible functions, and possesses both real and discrete valued latent variables. This Real and Discrete (Rad) approach retains the desirable normalizing flow properties of exact sampling, exact inference, and analytically computable probabilities, while at the same time allowing simultaneous modeling of both continuous and discrete structure in a data distribution.
中文速览
现有的流模型（flow-based model）只能处理连续数据，面对数据中天然存在的离散结构——比如多峰分布、类别标签或离散对称性——往往力不从心，强行用连续映射拟合这类结构会导致优化困难和数值不稳定。为此，文章提出了一种名为 Rad（Real and Discrete）的归一化流架构：核心思路是用分段可逆的满射函数把输入空间划分成若干不相交区域，每个区域对应一个离散潜变量，再对各区域内部应用标准的可逆变换，从而让潜变量同时包含连续分量和离散分量。这一设计保留了流模型的三大优良性质——精确采样、精确推断、对数似然可解析计算——同时通过递归堆叠形成深度混合模型，能够同时捕捉数据分布中的连续结构和离散结构。实验表明，Rad 在密度估计和生成任务上均优于或持平于现有连续流模型，为在流模型框架内显式建模离散结构提供了一条切实可行的路径。
原文 arXiv:1903.07714;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.07714v3