On Learning Rates and Schrödinger Operators
Bin Shi University of California, Berkeley. Email: Weijie J. Su The Wharton School, University of Pennsylvania. Email: Michael I. Jordan University of California, Berkeley. Email:
Abstract
The learning rate is perhaps the single most important parameter in the training of neural networks and, more broadly, in stochastic (nonconvex) optimization. Accordingly, there are numerous effective, but poorly understood, techniques for tuning the learning rate, including learning rate decay, which starts with a large initial learning rate that is gradually decreased. In this paper, we present a general theoretical analysis of the effect of the learning rate in stochastic gradient descent (SGD). Our analysis is based on the use of a learning-rate-dependent stochastic differential equation (lr-dependent SDE) that serves as a surrogate for SGD. For a broad class of objective functions, we establish a linear rate of convergence for this continuous-time formulation of SGD, highlighting the fundamental importance of the learning rate in SGD, and contrasting to gradient descent and stochastic gradient Langevin dynamics. Moreover, we obtain an explicit expression for the optimal linear rate by analyzing the spectrum of the Witten-Laplacian, a special case of the Schrödinger operator associated with the lr-dependent SDE. Strikingly, this expression clearly reveals the dependence of the
中文速览
学习率（learning rate）是神经网络训练中最关键的超参数之一，但"为什么学习率衰减（learning rate decay）有效"在理论上一直缺乏严谨解释。这篇论文以一个依赖学习率的随机微分方程（lr-dependent SDE）作为SGD的连续时间代理模型，从谱理论角度（具体是与Fokker-Planck方程关联的Witten-Laplacian算子的特征值分析）给出了SGD线性收敛速率的精确表达式。核心发现是：对于非凸目标函数，线性收敛速率 $\lambda_s$ 随学习率趋零而急剧趋零，而对于强凸函数，$\lambda_s$ 与学习率无关保持为常数——这一凸与非凸之间的本质差异，从数学上首次清晰解释了学习率衰减策略为何对深度学习（非凸优化）尤为关键：大学习率在训练初期带来快速收敛，随后衰减学习率则降低稳态误差。这项工作填补了非凸SGD理论分析中学习率效应的空白，也为学习率调度策略提供了有据可依的理论基础。
原文 arXiv:2004.06977;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.06977v1