A Divergence Minimization Perspective on Imitation Learning Methods
Seyed Kamyar Seyed Ghasemipour University of Toronto, Vector Institute、Richard Zemel University of Toronto, Vector Institute \ANDShixiang Gu Google Brain
Abstract
In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, due to multiple factors of variation, directly comparing these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective on IL algorithms based on divergence minimization. We present $f$ -MAX, an $f$ -divergence generalization of AIRL [1], a state-of-the-art IRL method. $f$ -MAX enables us to relate prior IRL methods such as GAIL [2] and AIRL [1], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches, and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings co
中文速览
模仿学习（Imitation Learning）领域长期存在一个令人困惑的现象：基于逆强化学习（IRL）的对抗方法在少量示范数据下远胜行为克隆（Behavioural Cloning，BC），但两者在理论最优点上其实都能还原专家策略，原因究竟何在？本文提出 f-MAX 框架，将 AIRL、GAIL、BC 等主流方法统一到「最小化策略分布与专家分布之间的 f-散度」这一概率视角下，揭示 AIRL 本质上是在最小化反向 KL 散度，并将 AIRL、GAIL 等方法都纳入同一代价正则化最大熵 IRL 框架。通过这一统一视角，作者在高维连续控制基准上做了系统消融实验，最终确认 IRL 优势的核心来源是「状态边际分布匹配」目标，而非奖励函数形式或散度类型的差别。基于此洞见，作者进一步将方法应用于无需专家示范、仅凭手工指定目标状态分布来训练多样化行为策略的「状态边际匹配」任务，在机械臂推物环境中验证了其有效性，为无奖励、无示范的策略学习提供了新思路。
原文 arXiv:1911.02256;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.02256v1