Generalization bounds for deep convolutional neural networks
Philip M. Long and Hanie Sedghi Google Brain Authors are ordered alphabetically.
Abstract
We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.
中文速览
卷积网络的泛化能力（generalization）究竟由什么决定？这篇论文专门为卷积神经网络（convolutional neural network, CNN）推导了泛化误差上界，核心思路是：网络的参数在训练过程中只需从初始值出发走"一小段距离"，而这段距离越短、网络参数越少，泛化能力就越有保证。与以往把卷积层简单等价成全连接层再套用旧结论不同，作者充分利用了卷积的权值共享结构，使得所得界与输入图像的像素数以及隐层特征图的尺寸完全无关；同时，新界对网络深度呈线性依赖，而非之前工作中常见的指数级增长，在理论上大幅收紧了估计。在 CIFAR-10 上的实验对比显示，不同超参数设置下，理论界与实际泛化差距的变化趋势基本吻合，验证了这些界的实用参考价值——这为理解"大模型为何仍能泛化"这一深度学习核心谜题提供了更贴近实际的理论支撑。
原文 arXiv:1905.12600;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.12600v6