Multi-fidelity Bayesian Optimisation with Continuous Approximations
Kirthevasan Kandasamy Gautam Dasarathy Jeff Schneider Barnabás Póczos
Abstract
Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, multi-fidelity methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. In many practical applications however, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data $N$ and/or few training iterations $T$ . Here, the approximations are best viewed as arising out of a continuous two dimensional space $(N,T)$ . In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiment
中文速览
连续近似空间下的黑盒优化一直缺乏系统的解决方案——调一个深度神经网络时，可以用更少的数据量和更少的训练轮次来"廉价模拟"最终性能，这两个维度共同构成了一个连续的保真度（fidelity）空间，而现有多保真度方法只能处理有限个离散近似。论文提出了BOCA（Bayesian Optimisation with Continuous Approximations），用高斯过程对目标函数和所有近似进行联合建模，让不同保真度之间自然共享信息，并借鉴GP-UCB的思路设计了一种"只有在低保真度不确定性已充分降低时才升级到更高保真度"的查询策略。理论上，BOCA被证明比完全忽略近似的方法获得更低的简单遗憾（simple regret），且避免了以往工作中对近似误差做均匀上界的强假设。在超参数调优和天体物理等真实任务上，BOCA在相同计算预算下明显优于多个基线，为那些拥有连续近似资源的昂贵优化场景提供了有理论保证的实用工具。
原文 arXiv:1703.06240;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.06240v1