Hyperparameter optimization with approximate gradient
Fabian Pedregosa Chaire Havas-Dauphine “Économie des Nouvelles Données” CEREMADE, CNRS UMR 7534, Université Paris-Dauphine, PSL Research University Département Informatique de l’École Normale Supérieure, Paris
Abstract
Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of $\ell_{2}$ -regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.
中文速览
机器学习模型中几乎都存在控制模型复杂度的超参数（hyperparameter），如何高效地选出最优超参数一直是个棘手问题。研究者提出了一种名为 HOAG（Hyperparameter Optimization with Approximate Gradient）的算法，核心思路是用"近似梯度"代替精确梯度来做基于梯度的超参数优化——每轮迭代时，内层模型参数和线性方程组都只需求解到一定精度，不必等到完全收敛就可以更新超参数，从而大幅节省计算开销。理论上，作者给出了该方法收敛到稳定点的充分条件，关键在于对近似误差序列的可求和性（summability）加以约束。在 ℓ₂ 正则化逻辑回归和核岭回归（kernel Ridge regression）的正则化参数选择实验中，HOAG 在精度和速度上均与当前最优方法持平甚至更优。这项工作的意义在于，它为大规模场景下的连续超参数调优提供了有理论保障且实用高效的解决方案。
原文 arXiv:1602.02355;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1602.02355v6