Partitioned Variational Inference: A unified framework encompassing federated and continual learning
Thang D. Bui University of Sydney, Australia; Cuong V. Nguyen University of Cambridge, UK; Siddharth Swaroop University of Cambridge, UK; Richard E. Turner University of Cambridge, UK;
Abstract
Variational inference (VI) has become the method of choice for fitting many modern probabilistic models. However, practitioners are faced with a fragmented literature that offers a bewildering array of algorithmic options. First, the variational family. Second, the granularity of the updates e.g. whether the updates are local to each data point and employ message passing or global. Third, the method of optimization (bespoke or blackbox, closed-form or stochastic updates, etc.). This paper presents a new framework, termed Partitioned Variational Inference (PVI), that explicitly acknowledges these algorithmic dimensions of VI, unifies disparate literature, and provides guidance on usage. Crucially, the proposed PVI framework allows us to identify new ways of performing VI that are ideally suited to challenging learning scenarios including federated learning (where distributed computing is leveraged to process non-centralized data) and continual learning (where new data and tasks arrive over time and must be accommodated quickly). We showcase these new capabilities by developing communication-efficient federated training of Bayesian neural networks and continual learning for Gaussian
中文速览
变分推断（Variational Inference, VI）是拟合概率模型的主流工具，但现有算法五花八门、文献分散，让实践者难以选择。这篇论文提出了一个统一框架——分区变分推断（Partitioned Variational Inference, PVI），把"用哪种变分族""更新粒度怎么定（逐数据点还是全局）""用什么优化方式"这三个核心维度明确梳理出来，将在线VI、消息传递、功率EP等大量看似无关的方法纳入同一套理论体系。更重要的是，PVI天然契合两类难题：联邦学习（Federated Learning，各客户端数据不可集中，需分布式训练）和持续学习（Continual Learning，数据或任务随时间流式到来），并在此基础上开发出通信高效的贝叶斯神经网络联邦训练方案，以及带私有伪点的高斯过程持续学习算法。实验表明，这些新方法在性能上显著超越现有最优方法，同时实现难度与标准VI相当，为概率机器学习在实际分布式和流式场景中的部署提供了切实可行的路径。
原文 arXiv:1811.11206;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.11206v1