Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, David Duvenaud University of Toronto and Vector Institute {wgrathwohl, choidami, ywu, roeder,
Abstract
Gradient-based optimization is the foundation of deep learning and reinforcement learning, but is difficult to apply when the mechanism being optimized is unknown or not differentiable. We introduce a general framework for learning low-variance, unbiased gradient estimators, applicable to black-box functions of discrete or continuous random variables. Our method uses gradients of a surrogate neural network to construct a control variate, which is optimized jointly with the original parameters. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.
中文速览
梯度优化在深度学习和强化学习中至关重要，但当目标函数不可微或包含离散随机变量时，传统反向传播便无从施展。本文提出 LAX/RELAX 框架，用一个可学习的神经网络作为"代理函数（surrogate）"来构造控制变量（control variate），将得分函数估计器（REINFORCE）与重参数化技巧结合，从而得到一个对任意黑箱函数均无偏、且方差更低的梯度估计器；代理网络的参数与模型参数同步优化，优化目标直接是梯度估计量的方差。实验表明，该方法在训练离散隐变量模型时显著优于此前的 REBAR 基线，并自然延伸出一种带有动作条件基线的无偏 Advantage Actor-Critic 强化学习算法。这项工作的意义在于，它将低方差无偏梯度估计从"需要知道 $f$ 的解析形式"的限制中解放出来，统一适用于离散变量模型、黑箱优化和强化学习等多种场景。
原文 arXiv:1711.00123;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.00123v3