Deep Convolutional Networks as shallow Gaussian Processes
Adrià Garriga-Alonso department of Engineering University of Cambridge、Carl Edward Rasmussen Department of Engineering University of Cambridge、Laurence Aitchison Department of Engineering University of Cambridge
Abstract
We show that the output of a (residual) \acCNN with an appropriate prior over the weights and biases is a \acGP in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a \acCNN, the equivalent kernel can be computed exactly and, unlike “deep kernels”, has very few parameters: only the hyperparameters of the original \acCNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original \acCNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for \acpGP with a comparable number of parameters. 111Code to replicate this paper is available at https://github.com/convnets-as-gps/convnets-as-gps
中文速览
卷积神经网络（CNN）在图像识别上效果出色，但做不确定性估计很难，而高斯过程（Gaussian Process，GP）恰好擅长精确的概率推断，只是缺乏CNN那样的模式识别能力。这篇论文证明了：当CNN（包括残差网络ResNet）每层的卷积滤波器数量趋向无穷时，网络的输出在数学上等价于一个GP，从而给出了CNN所对应的精确GP核函数。关键发现是，这个等价核的参数极少（只有原始CNN的超参数），而且可以高效计算——对一对图像求核值的代价，大约等于只用单个滤波器做一次前向传播；高效性来源于零均值独立权重使不同位置的贡献解耦，以及核的卷积结构允许同时并行计算所有位置的方差。在MNIST分类任务上，与32层ResNet等价的GP核在不做任何数据增强、不计算梯度的情况下达到0.84%错误率，创下同等参数规模GP方法的新纪录，为深度学习模型的严格贝叶斯推断提供了一条切实可行的路径。
原文 arXiv:1808.05587;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1808.05587v2