Variational Inference via χ𝜒\chi Upper Bound Minimization
Adji B. Dieng Columbia University、Dustin Tran Columbia University、Rajesh Ranganath Princeton University \ANDJohn Paisley Columbia University、David M. Blei Columbia University
Abstract
Variational inference (vi) is widely used as an efficient alternative to Markov chain Monte Carlo. It posits a family of approximating distributions $q$ and finds the closest member to the exact posterior $p$ . Closeness is usually measured via a divergence $D(q||p)$ from $q$ to $p$ . While successful, this approach also has problems. Notably, it typically leads to underestimation of the posterior variance. In this paper we propose chivi, a black-box variational inference algorithm that minimizes $D_{\chi}(p||q)$ , the $\chi$ -divergence from $p$ to $q$ . chivi minimizes an upper bound of the model evidence, which we term the $\chi$ upper bound (cubo). Minimizing the cubo leads to improved posterior uncertainty, and it can also be used with the classical vi lower bound (elbo) to provide a sandwich estimate of the model evidence. We study chivi on three models: probit regression, Gaussian process classification, and a Cox process model of basketball plays. When compared to expectation propagation and classical vi, chivi produces better error rates and more accurate estimates of posterior variance.
中文速览
变分推断（Variational Inference, VI）是贝叶斯后验近似中最常用的高效方法，但传统做法以 KL(q‖p) 为优化目标，会系统性地低估后验方差，在轻尾后验（如高斯过程分类）上表现尤差。本文提出 CHIVI 算法，改为最小化反向的 χ 散度 Dχ(p‖q)，并推导出对应的模型证据上界 CUBO；χ 散度具有"零回避"特性，迫使近似分布在后验有概率的地方也必须有概率，从而产生过散布（overdispersed）的近似，有效避免了方差低估问题。CHIVI 是黑盒算法，利用重参数化梯度对指数化上界做无偏蒙特卡洛优化，既保留上界性质，又无需针对具体模型推导；同时将 CUBO 与经典 ELBO 配合使用，可对模型证据形成"三明治夹逼"估计，为模型选择提供更可靠的依据。在 probit 回归、高斯过程分类和篮球比赛 Cox 过程三个实验中，CHIVI 相比传统 VI 和期望传播（EP）均取得了更低的分类错误率和更准确的后验方差估计，证明了这一方向修正带来的实质性收益。
原文 arXiv:1611.00328;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.00328v4