Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
Simon S. Du Machine Learning Department, School of Computer Science, Carnegie Mellon University. Email: Wei Hu Computer Science Department, Princeton University. Email: Jason D. Lee Department of Data Sciences and Operations, Marshall School of Business, University of Southern California. Email:
Abstract
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step size) effectively enforces the differences between squared norms across different layers to remain invariant without any explicit regularization. This result implies that if the weights are initially small, gradient flow automatically balances the magnitudes of all layers. Using a discretization argument, we analyze gradient descent with positive step size for the non-convex low-rank asymmetric matrix factorization problem without any regularization. Inspired by our findings for gradient flow, we prove that gradient descent with step sizes $\eta_{t}=O\left(t^{-\left(\frac{1}{2}+\delta\right)}\right)$ ( $0<\delta\leq\frac{1}{2}$ ) automatically balances two low-rank factors and converges to a bounded global optimum. Furthermore, for rank- $1$ asymmetric matrix factorization we give a finer analysis showing gradient descent with constant step size converges to the globa
中文速览
深度神经网络在训练时各层参数大小往往会自动趋于均衡，但背后的数学原因一直缺乏严格解释。这项工作证明了：对于带有线性、ReLU或Leaky ReLU激活函数的多层齐次模型（homogeneous models），梯度流（gradient flow，即步长无穷小的梯度下降）会精确保持相邻层权重矩阵Frobenius范数之差不变——这意味着只要初始化时各层参数量级相近，训练全程各层就会自动保持平衡，无需任何显式正则化。在此基础上，作者进一步分析了有限步长梯度下降在非凸的低秩不对称矩阵分解（low-rank asymmetric matrix factorization）问题上的行为，证明了采用适当衰减步长时梯度下降同样能维持两个因子的平衡并收敛至全局最优，对于秩为1的情形还给出了常数步长下全局线性收敛的精细分析。这一发现为理解深度学习中梯度下降的隐式正则化（implicit regularization）提供了严格的理论基础，也解释了为何不加任何约束的普通梯度下降在实践中能稳定收敛。
原文 arXiv:1806.00900;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.00900v2