Identifying Generalization Properties in Neural Networks
\nameHuan Wang, \nameNitish Shirish Keskar, \nameCaiming Xiong, \nameRichard Socher \addrSalesforce Research \email{huan.wang, nkeskar, cxiong,
Abstract
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order “smoothness” terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.
中文速览
训练好的深度网络为什么在"平坦"极小值处泛化更好、在"尖锐"极小值处更容易过拟合？这个经验现象虽然广为人知，却一直缺乏严格的理论解释。这篇文章借助 PAC-Bayes 框架，在对损失函数的黑塞矩阵（Hessian）满足局部 Lipschitz 连续性的温和假设下，从理论上证明了模型的泛化误差与黑塞矩阵的谱、刻画高阶光滑性的黑塞 Lipschitz 常数、参数的尺度以及训练样本数量共同相关。基于这一理论推导，作者提出了一个可量化泛化能力的评分指标（pacGen），并设计了一种利用黑塞矩阵估计来选择最优扰动幅度、进而优化扰动后模型的训练算法。这项工作填补了"为什么平坦极小值泛化更好"这一问题的理论空白，为设计更具泛化能力的深度模型训练方法提供了可解释的理论依据。
原文 arXiv:1809.07402;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.07402v1