Scalable Bayesian Optimization Using Deep Neural Networks
Jasper Snoek∗ Oren Rippel†∗ Kevin Swersky§ Ryan Kiros§ Nadathur Satish‡ Narayanan Sundaram‡ Md. Mostofa Ali Patwary‡ Prabhat⋆ Ryan P. Adams∗ ∗Harvard University, School of Engineering and Applied Sciences †Massachusetts Institute of Technology, Department of Mathematics §University of Toronto, Department of Computer Science ‡Intel Labs, Parallel Computing Lab ⋆NERSC, Lawrence Berkeley National Laboratory
Abstract
Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations. It relies on querying a distribution over functions defined by a relatively cheap surrogate model. An accurate model for this distribution over functions is critical to the effectiveness of the approach, and is typically fit using Gaussian processes (GPs). However, since GPs scale cubically with the number of observations, it has been challenging to handle objectives whose optimization requires many evaluations, and as such, massively parallelizing the optimization.
中文速览
贝叶斯优化（Bayesian optimization）在调超参数时很好用，但它通常依赖高斯过程（Gaussian process，GP）作为代理模型，而GP的计算复杂度随观测数量三次方增长，一旦需要大量评估或高度并行化就会卡住。为了解决这个瓶颈，作者提出用深度神经网络的最后一层隐层输出作为自适应基函数，再在其上接一个贝叶斯线性回归器，形成一套名为DNGO（Deep Networks for Global Optimization）的方法——神经网络负责学习灵活的非线性特征，贝叶斯线性回归负责量化不确定性，整体计算复杂度仅随观测数量线性增长。实验表明，DNGO在标准基准测试上与最先进的GP方法旗鼓相当，同时能够支撑大规模并行超参数搜索，成功将卷积网络在CIFAR-10和CIFAR-100上的错误率优化到6.37%和27.4%，并在图像描述生成任务中取得有竞争力的BLEU分数。这项工作的意义在于，它打破了贝叶斯优化难以扩展的瓶颈，让实际中训练成本高昂、超参数众多的大型深度学习模型也能享受到高效的自动调参。
原文 arXiv:1502.05700;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1502.05700v2