On Exact Computation with an Infinitely Wide Neural Net
Sanjeev Arora、Simon S. Du、Wei Hu、Zhiyuan Li Ruslan Salakhutdinov、Ruosong Wang Princeton University and Institute for Advanced Study. Email: for Advanced Study. Email: University. Email: University. Email: Mellon University. Mellon University. Email:
Abstract
How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its “width”— namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers — is allowed to increase to infinity? Such questions have come to the forefront in the quest to theoretically understand deep learning and its mysteries about optimization and generalization. They also connect deep learning to notions such as Gaussian processes and kernels. A recent paper (Jacot et al., 2018) introduced the Neural Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in the infinite width limit trained by gradient descent; this object was implicit in some other recent papers. An attraction of such ideas is that a pure kernel-based method is used to capture the power of a fully-trained deep net of infinite width.
中文速览
卷积神经网络宽度趋于无穷时，其完整训练行为可以被一个叫做"神经正切核"（Neural Tangent Kernel，NTK）的固定核函数精确刻画，但此前对卷积版本（CNTK）的精确计算被认为不可行，研究者只能依赖蒙特卡洛近似，导致性能损失严重。本文提出了首个高效的动态规划算法，能在 GPU 上精确计算带池化操作的 CNTK，从而将无限宽卷积网络的能力真正落地为一个纯核方法。在 CIFAR-10 上，对应11层卷积网络的 CNTK 达到了77%的分类准确率，比此前最优核方法高出约10%，与同结构有限宽有监督深度网络的差距仅6%。此外，论文还给出了首个非渐近的严格证明，确认足够宽的完整训练网络确实收敛到 NTK 核回归预测器，为理论理解深度学习的优化与泛化行为提供了坚实基础。
原文 arXiv:1904.11955;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.11955v2