Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
Roman Novak †, Lechao Xiao † , Jaehoon Lee ‡ Yasaman Bahri ‡ Greg Yang∘, Jiri Hron⋄, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein Google Brain, ∘Microsoft Research AI, ⋄ Department of Engineering, University of Cambridge {romann, xlc, jaehlee, yasamanb, danabo, jpennin, Google AI Residents (g.co/airesidency). ,†‡{}^{\dagger},{}^{\ddagger} Equal contribution.
Abstract
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible.
中文速览
宽通道卷积神经网络（CNN）在完全贝叶斯训练下，其行为等价于一个高斯过程（Gaussian Process，GP）——这篇论文从理论上严格证明了这一点，并将此前仅适用于全连接网络的等价关系推广到了含或不含池化层的多层CNN。最关键的发现是：在没有池化层的情况下，有权重共享（即标准CNN）与无权重共享（局部连接网络）所对应的GP完全相同，这意味着CNN所依赖的平移等变性在无限通道的贝叶斯极限中根本不起作用——这是CNN与全连接网络之间一个本质性的理论差异。作者还提出了一种蒙特卡洛方法来估计含池化层等解析形式过于复杂时的GP核，并在CIFAR-10上取得了当时无可训练核GP的最优结果。实验同时表明，经过精心调参的有限通道CNN用SGD训练可以显著超越对应的GP，说明SGD训练相比完全贝叶斯推断仍有其独特优势，这为理解深度学习中归纳偏置的来源提供了重要的理论视角。
原文 arXiv:1810.05148;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.05148v4