Sandwiching the marginal likelihood using bidirectional Monte Carlo
Roger B. Grosse University of Toronto Zoubin Ghahramani University of Cambridge Ryan P. Adams Twitter and Harvard University
Abstract
Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one’s ML estimates. We present bidirectional Monte Carlo, a technique for obtaining accurate log-ML estimates on data simulated from a model. This method obtains stochastic lower bounds on the log-ML using annealed importance sampling or sequential Monte Carlo, and obtains stochastic upper bounds by running these same algorithms in reverse starting from an exact posterior sample. The true value can be sandwiched between these two stochastic bounds with high probability. Using the ground truth log-ML estimates obtained from our method, we quantitatively evaluate a wide variety of existing ML estimators on several latent variable models: clustering, a low rank approximation, and a binary attributes model. These experiments yield insights into how to accurately estimate marginal likelihoods.
中文速览
边际似然（marginal likelihood）是贝叶斯模型选择的核心指标，但对大多数模型来说它根本算不出来，而且现有的各种近似估计方法到底准不准，几乎没有办法客观检验。作者提出了"双向蒙特卡洛"（bidirectional Monte Carlo）方法：先用退火重要性采样（annealed importance sampling, AIS）或序贯蒙特卡洛（sequential Monte Carlo, SMC）从先验跑到后验，得到对数边际似然的随机下界；再从精确后验样本出发把同样的算法反向运行，得到随机上界；真实值就被高概率地夹在这两个界之间，计算量越大两个界收得越紧。借助这把"卡尺"，作者在聚类、低秩分解、二值属性三类隐变量模型的模拟数据上，对市面上几乎所有主流的边际似然估计器进行了定量横评，揭示了哪些方法系统性地高估或低估、以及如何在实践中选用可靠的估计策略，为整个领域提供了一个严格的评测基准。
原文 arXiv:1511.02543;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.02543v1