Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
Hadi S. Jomaa 11 Josif Grabocka 11 Lars Schmidt-Thieme 11
Abstract
Hyperparameter tuning is an omnipresent problem in machine learning as it is an integral aspect of obtaining the state-of-the-art performance for any model. Most often, hyperparameters are optimized just by training a model on a grid of possible hyperparameter values and taking the one that performs best on a validation sample (grid search). More recently, methods have been introduced that build a so-called surrogate model that predicts the validation loss for a specific hyperparameter setting, model and dataset and then sequentially select the next hyperparameter to test, based on a heuristic function of the expected value and the uncertainty of the surrogate model called acquisition function (sequential model-based Bayesian optimization, SMBO). In this paper we model the hyperparameter optimization problem as a sequential decision problem, which hyperparameter to test next, and address it with reinforcement learning. This way our model does not have to rely on a heuristic acquisition function like SMBO, but can learn which hyperparameters to test next based on the subsequent reduction in validation loss they will eventually lead to, either because they yield good models themselve
中文速览
训练机器学习模型时，超参数（hyperparameter）的选择对最终性能至关重要，而现有方法要么靠穷举网格搜索，要么依赖贝叶斯优化中人工设计的"采集函数"来决定下一步试哪组超参数。本文把超参数搜索重新建模为一个序贯决策问题，用强化学习（reinforcement learning, RL）训练出一个策略网络 Hyp-RL，让智能体在超参数空间里自主探索——它既会因为找到好的超参数配置而获得奖励，也会因为探索帮助自己建立更准确的"地图"而在后续步骤中做出更优选择，从而彻底绕开了人工设计采集函数的局限。在涵盖 50 个 UCI 分类数据集的大规模实验中，Hyp-RL 的表现持续优于当前最先进的超参数优化方法。这项工作首次将超参数调优严格纳入强化学习框架，为构建无需人工干预、可跨数据集迁移经验的自动调参系统提供了新思路。
原文 arXiv:1906.11527;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.11527v1