Gradient Estimation Using Stochastic Computation Graphs
John Schulman1,2、Nicolas Heess1 \AND Theophane Weber1、Pieter Abbeel2 \AND 1 Google DeepMind、2 University of California, Berkeley, EECS Department
Abstract
In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external world. Estimating the gradient of this loss function, using samples, lies at the core of gradient-based learning algorithms for these problems. We introduce the formalism of stochastic computation graphs—directed acyclic graphs that include both deterministic functions and conditional probability distributions—and describe how to easily and automatically derive an unbiased estimator of the loss function’s gradient. The resulting algorithm for computing the gradient estimator is a simple modification of the standard backpropagation algorithm. The generic scheme we propose unifies estimators derived in variety of prior work, along with variance-reduction techniques therein. It could assist researchers in developing intricate models involving a combination of stochastic and deterministic operations, enabling, for example, attention, memory, and control actions.
中文速览
随着神经网络向概率化、决策化方向发展，研究者越来越需要对"期望损失"（loss defined by expectation）求梯度，但经典反向传播算法只能处理纯确定性计算图，无法直接应对含随机节点的模型。本文引入"随机计算图"（stochastic computation graph）这一统一形式框架，将确定性函数节点与随机变量节点整合进同一个有向无环图中，并从中系统推导出无偏梯度估计器——它把经典的得分函数估计器（score function / REINFORCE）和路径导数估计器（pathwise derivative / reparameterization trick）统一为同一套规则的特例。关键发现是：只需构造一个"替代损失"（surrogate loss）函数，对它运行标准反向传播即可自动得到正确的梯度估计，相当于在随机节点处插入额外的梯度信号，实现上几乎零门槛。这项工作不仅将强化学习中的策略梯度与变分推断中的重参数化技巧等先前各自独立的方法纳入同一理论体系，还为研究者构建含注意力、记忆、控制动作等随机与确定性操作混合的复杂模型提供了开箱即用的自动微分支持。
原文 arXiv:1506.05254;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1506.05254v3