SGDR: Stochastic Gradient Descent with Warm Restarts
Ilya Loshchilov、Frank Hutter University of Freiburg Freiburg, Germany,
Abstract
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR
中文速览
训练深度神经网络时，学习率调度策略直接影响收敛速度和最终精度，现有的阶梯式衰减方案往往需要耗费大量训练轮次才能达到理想效果。本文提出了一种带热重启的随机梯度下降方法（SGDR），核心思路是用余弦退火周期性地将学习率从较高值平滑降至极小值，然后突然"重启"回高学习率，并可通过倍增周期长度来兼顾早期的快速收敛与后期的精细优化。在 CIFAR-10 和 CIFAR-100 基准测试中，SGDR 比传统调度方案少用 2 到 4 倍的训练轮次就能达到相当甚至更好的精度，将两个数据集的最优错误率分别刷新至 3.14% 和 16.21%；在脑电信号和缩小版 ImageNet 数据集上也展现出同样的优势。这项工作以极低的实现代价显著提升了深度网络训练的效率，其余弦退火与热重启思想此后被广泛采用，成为深度学习学习率调度的重要基础方法之一。
原文 arXiv:1608.03983;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.03983v5