Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
Wenlong Mou Key Laboratory of Machine Perception, School of EECS, Peking University Liwei Wang Key Laboratory of Machine Perception, School of EECS, Peking University Xiyu Zhai School of Mathematics, University of Science and Technology of China Kai Zheng Key Laboratory of Machine Perception, School of EECS, Peking University
Abstract
Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex learning problems not only have important theoretical consequences, but are also critical to generalization errors of deep learning.
中文速览
训练轮数少就能泛化好，这个直觉在非凸深度学习中一直缺乏严格的理论支撑——随机梯度朗之万动力学（Stochastic Gradient Langevin Dynamics, SGLD）在每步梯度更新时注入高斯噪声，本文首次对非凸目标函数下的SGLD给出了有实际意义的、依赖于算法本身的泛化误差上界。研究分别采用一致稳定性（uniform stability）和PAC-贝叶斯（PAC-Bayesian）两种框架推导出离散时间、非渐近的界：前者给出O(1/n)的快速收敛率，关键量是累积步长的平方根；后者给出O(1/√n)的界，但通过ℓ²正则化使早期大步长的贡献指数衰减，且将统一Lipschitz常数替换为轨迹上的实际梯度模。两个结果均不依赖参数空间维度或参数范数等经典容量度量，从理论上严格刻画了"训练越快、泛化越好"这一深度学习中的普遍经验现象，为理解随机梯度方法在复杂模型中的统计学习性质提供了重要的理论基础。
原文 arXiv:1707.05947;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.05947v1