Three Mechanisms of Weight Decay Regularization
Guodong Zhang, Chaoqi Wang, Bowen Xu, Roger Grosse University of Toronto, Vector Institute {gdzhang, cqwang, bowenxu,
Abstract
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of $L_{2}$ regularization. Literal weight decay has been shown to outperform $L_{2}$ regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks.
中文速览
权重衰减（weight decay）是训练神经网络时最常用的正则化手段之一，但人们对它"为什么有效"的理解一直模糊，尤其是在批归一化（Batch Normalization）和自适应优化器普及之后，传统的"它就是L₂正则化"这一解释已经站不住脚。这篇论文针对SGD、Adam、K-FAC三种优化器，在多种网络结构和数据集上系统地做了对比实验，发现权重衰减总是优于直接加L₂惩罚项，并归纳出三种截然不同的起效机制：在带有批归一化的网络里，权重衰减压低了权重的尺度，从而提高了等效学习率，放大了梯度噪声的正则化效果；在K-FAC-G（高斯-牛顿度量版K-FAC）中，权重衰减等价于对网络输入-输出雅可比矩阵（Jacobian）的F范数施加惩罚，而这正是已知能改善泛化的手段；在实际使用K-FAC时，阻尼项对权重尺度敏感，若不加权重衰减权重会膨胀，阻尼项占主导后K-FAC退化为一阶优化器，权重衰减通过控制权重尺度让K-FAC保持二阶特性。这一发现揭示了"同一个超参数在不同设置下作用原理完全不同"，为更有针对性地设计神经网络正则化策略提供了具体的理论依据。
原文 arXiv:1810.12281;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.12281v1