Deep Neural Networks as Gaussian Processes
Jaehoon Lee , Yasaman Bahri , Roman Novak , Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein Google Brain {jaehlee, yasamanb, romann, schsam, jpennin, Both authors contributed equally to this work.Work done as a member of the Google AI Residency program (g.co/airesidency).
Abstract
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network.
中文速览
无限宽深度神经网络（deep neural network）在参数服从独立同分布先验时，等价于一个高斯过程（Gaussian process，GP）——这篇论文从理论上严格推导了这一等价关系，并给出了一套递归计算对应GP核函数（kernel function）的高效方法。有了这个"神经网络GP"（NNGP），就可以对宽深度网络直接做精确贝叶斯推断，而无需传统的梯度训练。作者在MNIST和CIFAR-10上验证发现：随着网络宽度增大，梯度训练网络的精度逐渐逼近对应GP的精度；GP的预测不确定性与模型出错的位置高度吻合；在许多设置下，GP预测表现甚至优于有限宽度的训练网络。这项工作在理论上打通了深度神经网络与贝叶斯非参数方法之间的桥梁，为理解深度网络的泛化行为提供了一个新的分析视角。
原文 arXiv:1711.00165;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.00165v3