Bayesian Optimization for Iterative Learning
Vu Nguyen University of Oxford Sebastian Schulze ††University of Oxford Michael A. Osborne University of Oxford These authors contributed equally.
Abstract
The performance of deep (reinforcement) learning systems crucially depends on the choice of hyperparameters. Their tuning is notoriously expensive, typically requiring an iterative training process to run for numerous steps to convergence. Traditional tuning algorithms only consider the final performance of hyperparameters acquired after many expensive iterations and ignore intermediate information from earlier training steps. In this paper, we present a Bayesian optimization (BO) approach which exploits the iterative structure of learning algorithms for efficient hyperparameter tuning. We propose to learn an evaluation function compressing learning progress at any stage of the training process into a single numeric score according to both training success and stability. Our BO framework is then balancing the benefit of assessing a hyperparameter setting over additional training steps against their computation cost. We further increase model efficiency by selectively including scores from different training steps for any evaluated hyperparameter set. We demonstrate the efficiency of our algorithm by tuning hyperparameters for the training of deep reinforcement learning agents and c
中文速览
深度学习与深度强化学习的训练效果对超参数极其敏感，但每次评估一组超参数都要跑完整个训练流程，代价极高。现有贝叶斯优化（Bayesian Optimization, BO）方法通常只看训练末尾的平均表现，白白丢弃了中间每一步的训练曲线信息。本文提出 BOIL 方法，用一个可从数据中自动学习的 Sigmoid 压缩函数把整条训练曲线压缩成单一评分，同时兼顾训练成功率与稳定性，再结合主动学习式数据增强技术充分利用中间步骤的信息，让高斯过程代理模型在不损失精度的前提下保持良好数值条件。在多个深度强化学习基准和卷积神经网络调参实验中，BOIL 在相同挂钟时间内找到最优超参数的能力全面超越所有对比基线，证明了充分挖掘迭代训练结构能够显著提升超参数调优效率。
原文 arXiv:1909.09593;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.09593v5