Generative Adversarial Imitation Learning
Jonathan Ho Stanford University、Stefano Ermon Stanford University
Abstract
Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert’s cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning. We show that a certain instantiation of our framework draws an analogy between imitation learning and generative adversarial networks, from which we derive a model-free imitation learning algorithm that obtains significant performance gains over existing model-free methods in imitating complex behaviors in large, high-dimensional environments.
中文速览
模仿学习领域长期存在一个痛点：想让智能体（agent）从专家示范中学会行为，传统做法要先用逆强化学习（inverse reinforcement learning, IRL）反推出专家的"代价函数"，再用强化学习在这个函数上训练策略，流程繁琐且计算极为昂贵。这篇论文从理论上证明，上述"IRL + RL"两步流程本质上等价于让学习策略的状态-动作访问分布（occupancy measure）去匹配专家的分布，并据此提出了一个可以直接从数据中提取策略的统一框架，跳过中间的代价函数学习步骤。在此框架下，作者推导出一种具体算法——生成对抗模仿学习（Generative Adversarial Imitation Learning, GAIL），它将模仿学习与生成对抗网络（generative adversarial networks, GAN）类比：判别器负责区分专家与学习者的行为分布，策略网络则不断尝试"骗过"判别器，两者交替对抗训练。在多个高维连续控制的物理仿真任务上，GAIL 以少量专家数据就大幅超越了此前的无模型模仿学习方法，为在复杂环境中高效复现专家行为提供了一条既有理论保障又实用的新路径。
原文 arXiv:1606.03476;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.03476v1