Gaussian Process Behaviour in Wide Deep Neural Networks
Alexander G. de G. Matthews \addrDepartment of Engineering Trumpington Street University of Cambridge, UK \ANDMark Rowland \addrDepartment of Pure Mathematics and Mathematical Statistics Wilberforce Road University of Cambridge, UK \ANDJiri Hron \addrDepartment of Engineering Trumpington Street University of Cambridge, UK \ANDRichard E. Turner \addrDepartment of Engineering Trumpington Street University of Cambridge, UK \ANDZoubin Ghahramani \addrDepartment of Engineering Trumpington Street University of Cambridge, UK Uber AI Labs
Abstract
Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.111Code for the experiments in the paper can be found at https://github.com/widedeepnetworks/widedeepnetworks
中文速览
随着神经网络层数加深，人们一直想搞清楚"宽网络在概率意义上究竟收敛到什么"这一理论问题。论文证明了：当一个多隐藏层的全连接前馈神经网络的每一层宽度都趋于无穷时，网络所定义的随机函数会依分布收敛到一个高斯过程（Gaussian Process, GP），其核函数由一套递归公式给出——这将Neal（1996）对单隐藏层网络的经典结论严格推广到了任意固定深度的深层网络，且层宽允许以不同速率同时增长。为了衡量有限网络与极限高斯过程之间的差距，作者用最大均值差异（Maximum Mean Discrepancy, MMD）进行了实验评估，并在六个真实数据集上用"黄金标准"MCMC方法对有限贝叶斯深度网络做推断，再与对应高斯过程的精确推断结果逐一比对，发现其中五个数据集上两者吻合得相当紧密。这一结果意味着：许多打着贝叶斯深度学习旗号的实验，其效果很可能用一个合适核函数的高斯过程就能复现，因此为贝叶斯深度学习社区提供了一个重要的基准参照，同时也提示研究者在不希望网络退化为高斯过程时，需要有意识地采用非高斯替代架构。
原文 arXiv:1804.11271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.11271v2