A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
Chelsea Finn∗, Paul Christiano , Pieter Abbeel, Sergey Levine University of California, Berkeley Indicates equal contribution.
Abstract
Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously learned by a discriminator. While the idea of learning cost functions is relatively new to the field of generative modeling, learning costs has long been studied in control and reinforcement learning (RL) domains, typically for imitation learning from demonstrations. In these fields, learning the cost function underlying observed behavior is known as inverse reinforcement learning (IRL) or inverse optimal control. While at first the connection between cost learning in RL and cost learning in generative modeling may appear to be a superficial one, we show in this paper that certain IRL methods are in fact mathematically equivalent to GANs. In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator’s density can be evaluated and is provided as an additional input to the discriminator. Interestingly, maximum entropy IRL is a special case of an energy-based model. We discuss the interpretation of GANs as an algorithm for training energ
中文速览
生成对抗网络（GAN）和逆强化学习（IRL）看似来自完全不同的领域，但这篇论文从数学上严格证明了两者其实是等价的：基于采样的最大熵逆强化学习（Maximum Entropy IRL）算法，与一种判别器能够利用生成器密度信息的GAN，在梯度更新层面完全等价。作者进一步指出，最大熵IRL本身是基于能量的模型（Energy-Based Model，EBM）的特例，由此将GAN、IRL、EBM三个领域统一在同一框架下。这一等价关系带来了实际启发：即使生成器的概率密度可以精确计算（通常此时会直接做最大似然估计），对抗式训练依然有优势——它能让生成器专注于"填充"高概率区域而非"覆盖"所有模式，从而产生更真实的样本，这与IRL相比于行为克隆（behavioral cloning）的优势如出一辙。这项工作的重要价值在于，它为三个社区之间的算法互借提供了理论基础，有望推动更稳定、更可扩展的训练方法的开发——这恰恰是三个领域共同面临的核心挑战。
原文 arXiv:1611.03852;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.03852v3