SAME but Different: Fast and High-Quality Gibbs Parameter Estimation
Huasha Zhao Computer Science Division UC Berkeley Berkeley, CA 94720、Biye Jiang Computer Science Division UC Berkeley Berkeley, CA 94720 \ANDJohn Canny Computer Science Division UC Berkeley Berkeley, CA 94720
Abstract
Gibbs sampling is a workhorse for Bayesian inference but has several limitations when used for parameter estimation, and is often much slower than non-sampling inference methods. SAME (State Augmentation for Marginal Estimation) [14, 8] is an approach to MAP parameter estimation which gives improved parameter estimates over direct Gibbs sampling. SAME can be viewed as cooling the posterior parameter distribution and allows annealed search for the MAP parameters, often yielding very high quality (lower loss) estimates. But it does so at the expense of additional samples per iteration and generally slower performance. On the other hand, SAME dramatically increases the parallelism in the sampling schedule, and is an excellent match for modern (SIMD) hardware. In this paper we explore the application of SAME to graphical model inference on modern hardware. We show that combining SAME with factored sample representation (or approximation) gives throughput competitive with the fastest symbolic methods, but with potentially better quality. We describe experiments on Latent Dirichlet Allocation, achieving speeds similar to the fastest reported methods (online Variational Bayes) and lower c
中文速览
吉布斯采样（Gibbs sampling）在贝叶斯参数估计中收敛慢、难以找到MAP最优解，而变分贝叶斯等符号方法虽然快，但估计质量有时欠佳。本文将SAME（State Augmentation for Marginal Estimation，状态增广边缘估计）与因子化样本表示相结合，通过复制潜变量来"冷却"参数后验分布，使采样自然向最优解集中，同时大幅提升了并行度以充分利用现代SIMD硬件。将该方法应用于隐含狄利克雷分配（LDA）的实验表明，其速度与目前最快的在线变分贝叶斯方法相当，而交叉验证损失却低于其他LDA实现。这一结果说明，SAME方法兼顾了采样方法的高质量估计与符号方法的高速度，且实现简单，有望成为多种概率图模型推断的首选方案。
原文 arXiv:1409.5402;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1409.5402v1