Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Arthur Jacot École Polytechnique Fédérale de Lausanne、Franck Gabriel Imperial College London and École Polytechnique Fédérale de Lausanne、Clément Hongler École Polytechnique Fédérale de Lausanne
Abstract
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (16; 4; 7; 13; 6), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_{\theta}$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial.
中文速览
训练深度神经网络时，损失曲面非凸、优化行为难以理论分析，一直是悬而未决的难题。这篇文章引入了一个新工具——神经切线核（Neural Tangent Kernel，NTK），证明了在梯度下降过程中，网络输出函数的变化方向等价于以NTK为核的核梯度下降，从而把参数空间里的非凸优化问题转化为函数空间里的凸问题。关键发现是：当网络宽度趋向无穷时，NTK在初始化时收敛到一个确定性的极限核，并且在整个训练过程中保持不变，使得网络训练动态可以用一个线性微分方程精确刻画。由此得到的推论包括：训练收敛性等价于极限NTK的正定性（作者在数据分布于球面时给出了严格证明），最快收敛的方向对应NTK的最大核主成分，这也为早停（early stopping）提供了理论依据。这项工作首次从理论上严格描述了深度网络训练的完整动力学，将神经网络与核方法统一在同一框架下，为理解神经网络的泛化能力和优化行为打开了新的视角。
原文 arXiv:1806.07572;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.07572v4