Neural Optimizer Search with Reinforcement Learning
Irwan Bello Barret Zoph Vijay Vasudevan Quoc V. Le
Abstract
We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures. We train a Recurrent Neural Network controller to generate a string in a domain specific language that describes a mathematical update equation based on a list of primitive functions, such as the gradient, running average of the gradient, etc. The controller is trained with Reinforcement Learning to maximize the performance of a model after a few epochs. On CIFAR-10, our method discovers several update rules that are better than many commonly used optimizers, such as Adam, RMSProp, or SGD with and without Momentum on a ConvNet model. We introduce two new optimizers, named PowerSign and AddSign, which we show transfer well and improve training on a variety of different tasks and architectures, including ImageNet classification and Google’s neural machine translation system.
中文速览
手工设计深度学习优化器（如Adam、SGD）费时费力，且受限于人类经验，这篇论文尝试用机器自动发现更好的优化算法。研究者训练了一个循环神经网络（RNN）作为"控制器"，让它用强化学习在一套专门设计的数学描述语言里自动组合梯度、移动平均等基本运算，生成新的参数更新公式，再用这些公式在小模型上跑几轮的准确率作为奖励信号来反复改进控制器。在CIFAR-10上搜索到的更新规则普遍优于Adam、RMSProp和带/不带动量的SGD，其中两个被命名为PowerSign和AddSign的新优化器迁移性很强——在ImageNet图像分类上提升了约0.4%的准确率，在谷歌神经机器翻译系统上也带来了最高0.7 BLEU的提升。这项工作的意义在于证明了"让AI来发明优化算法"这条路可行，而且搜到的公式简洁、内存占用低，容易直接用于不同架构和任务。
原文 arXiv:1709.07417;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.07417v2