A Convergence Theory for Deep Learning via Over-Parameterization
Zeyuan Allen-Zhu Microsoft Research AI Yuanzhi Li Stanford University Princeton University Zhao Song UT-Austin University of Washington Harvard University
Abstract
Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely unsettled.
中文速览
深度神经网络在实践中表现出色，但为什么用随机梯度下降（SGD）从随机初始化出发就能高效找到全局最优解，理论上一直没有令人满意的解释，尤其是对多层网络而言。本文证明：只要网络足够"过参数化"（即每层神经元数量是层数L和样本量n的多项式量级），并且训练数据彼此不重复，SGD就能在多项式时间内找到深度神经网络的全局最优解。核心技术洞察在于，在随机初始化的足够大邻域内，优化曲面近似凸且半光滑，这使得有限宽度的过参数化网络等价于神经切核（Neural Tangent Kernel, NTK），从而绕开了ReLU激活函数带来的非凸非光滑难题。理论结果覆盖全连接网络、卷积网络和残差网络，且运行时间对层数L的依赖仅为多项式而非指数级，为理解深层网络的训练机制提供了迄今最具普遍性的理论保障。
原文 arXiv:1811.03962;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.03962v5