Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
\nameYann Ollivier \addrCNRS、LRI (UMR 8623), Université Paris-Saclay 91405 Orsay, France \AND\nameLudovic Arnold \addrUniv. Paris-Sud, LRI 91405 Orsay, France \AND\nameAnne Auger \nameNikolaus Hansen \addrInria、CMAP, Ecole polytechnique 91128 Palaiseau, France
Abstract
We present a canonical way to turn any smooth parametric family of probability distributions on an arbitrary search space $X$ into a continuous-time black-box optimization method on $X$ , the information-geometric optimization (IGO) method. Invariance as a major design principle keeps the number of arbitrary choices to a minimum. The resulting IGO flow is the flow of an ordinary differential equation conducting the natural gradient ascent of an adaptive, time-dependent transformation of the objective function. It makes no particular assumptions on the objective function to be optimized.
中文速览
黑箱优化领域长期缺乏一套统一、内在自洽的理论框架——现有算法要么依赖启发式规则，要么无法同时满足对搜索空间坐标变换、分布参数化方式以及目标函数单调变换的不变性。本文提出信息几何优化（Information-Geometric Optimization，IGO）方法：以任意搜索空间上的参数化概率分布族为出发点，利用Fisher信息矩阵定义的自然梯度，对经过分位数自适应变换后的目标函数做连续时间梯度上升，从而导出一条具有严格数学意义的微分方程流，再通过Euler离散化得到可实际运行的优化算法。理论分析证明IGO流同时满足三重不变性，并在数学上统一了多个经典算法：对高斯分布族可复现CMA-ES与xNES，对Bernoulli分布族可复现PBIL与cGA，交叉熵方法也作为大步长极限自然浮现；此外，框架还为受限玻尔兹曼机（Restricted Boltzmann Machine，RBM）分布族导出了一种全新的离散优化算法，初步实验表明其能在单次运行中同时探索多个最优解。这一工作为设计新的黑箱优化算法提供了原则性路径，使算法行为真正由问题的内禀几何结构决定，而非任意的编码或参数化选择。
原文 arXiv:1106.3708;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1106.3708v4