SALR: Sharpness-aware Learning Rate Scheduler for Improved Generalization
Xubo Yue Industrial and Operations Engineering University of Michigan, Ann Arbor Maher Nouiehed American University of Beirut, Lebanon Raed Al Kontar Industrial and Operations Engineering University of Michigan, Ann Arbor
Abstract
In an effort to improve generalization in deep learning and automate the process of learning rate scheduling, we propose SALR: a sharpness-aware learning rate update technique designed to recover flat minimizers. Our method dynamically updates the learning rate of gradient-based optimizers based on the local sharpness of the loss function. This allows optimizers to automatically increase learning rates at sharp valleys to increase the chance of escaping them. We demonstrate the effectiveness of SALR when adopted by various algorithms over a broad range of networks. Our experiments indicate that SALR improves generalization, converges faster, and drives solutions to significantly flatter regions.
中文速览
训练深度神经网络时，模型往往会陷入"尖锐"的局部极小值，导致泛化能力差，而且学习率的调节全靠人工经验，费时费力。为此，作者提出了 SALR（Sharpness-Aware Learning Rate，锐度感知学习率）：在训练过程中实时估计当前参数所在区域的"尖锐度"，如果检测到尖锐山谷就自动调大学习率帮助逃离，落入平坦区域则调小学习率以稳定收敛，整个过程无需人工设定学习率计划表。这一机制可以无缝插入 SGD、Adam、Entropy-SGD 等主流优化器，在图像分类、文本预测、预训练微调等多项任务上均带来了泛化性能的提升，同时收敛更快、找到的解也显著更平坦。这项工作的价值在于，它把"寻找平坦极小值"这一对泛化至关重要的目标直接编码进学习率机制里，既有理论保证又有广泛的实用性，为自动化学习率调度提供了一条新思路。
原文 arXiv:2011.05348;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.05348v2