PAC-Bayes with Backprop
Omar Rivasplata DeepMind Vikram Tankasali DeepMind Csaba Szepesvari DeepMind
Abstract
We explore the family of methods ‘PAC-Bayes with Backprop’ (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated on MNIST and on various UCI data sets. Our experiments show two striking observations: we obtain competitive test set error estimates ( $\sim 1.4\%$ on MNIST) and at the same time we compute non-vacuous bounds with much tighter values ( $\sim 2.3\%$ on MNIST) than previous results. These observations suggest that neural nets trained by PBB may lead to self-bounding learning, where the available data can be used to simultaneously learn a predictor and certify its risk, with no need to follow a data-splitting protocol.
中文速览
用反向传播直接最小化PAC-Bayes上界，可以同时训练一个概率神经网络并为其泛化误差提供可证明的保证——但此前的工作要么牺牲预测精度，要么得到的上界过于宽松、几乎没有实际意义。本文提出了"PAC-Bayes with Backprop"（PBB）框架下的两个训练目标，一个来自已有的PAC-Bayes-λ界，另一个来自作者新推导的PAC-Bayes二次界，两者均采用经典固定先验和普通SGD优化，无需任何特殊技巧。在MNIST上，这两个目标同时实现了约1.4%的测试错误率和约2.3%的风险上界，不仅预测精度与Blundell等人的Bayes By Backprop方法相当，还将风险证书与实测误差之间的差距大幅收窄至前人工作的一小部分。这一结果表明，概率神经网络有望实现"自界学习"——用同一批数据既学出预测器，又为其在未见数据上的表现提供可信的误差保证，无需额外划分数据集。
原文 arXiv:1908.07380;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1908.07380v5