Task-Relevant Adversarial Imitation Learning
Konrad Żołna1,2,、Scott Reed2,∗、Alexander Novikov2、Sergio Gómez Colmenarejo2、David Budden2、Serkan Cabi2、Misha Denil2、Nando de Freitas2、Ziyu Wang2 \AND1Jagiellonian University 2DeepMind Equal contribution. Work done at DeepMind. Corresponding authors:
Abstract
We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performance. We analyze this problem in detail and propose a solution that outperforms standard Generative Adversarial Imitation Learning (GAIL). Our proposed method, Task-Relevant Adversarial Imitation Learning (TRAIL), uses constrained discriminator optimization to learn informative rewards. In comprehensive experiments, we show that TRAIL can solve challenging robotic manipulation tasks from pixels by imitating human operators without access to any task rewards, and clearly outperforms comparable baseline imitation agents, including those trained via behaviour cloning and conventional GAIL.
中文速览
模仿学习中有一个被忽视的核心缺陷：判别器（discriminator）网络会把与任务无关的视觉特征（比如亮度、初始姿态）和"专家行为"错误地绑定在一起，导致它给智能体的奖励信号越来越没用，训练最终崩溃。针对这一问题，作者提出了"任务相关对抗式模仿学习"（Task-Relevant Adversarial Imitation Learning，TRAIL）：在训练判别器时额外引入一组"约束集合"——这些观测数据只含无关特征、不含任务信息——并要求判别器无法区分专家与智能体在这组数据上的表现，从而逼迫它把注意力集中到真正与任务相关的特征上。在多个基于像素输入的仿真机械臂操作任务上，TRAIL无需任何人工设计的奖励函数，仅凭100条人类示范就能完成抓取、插块等复杂操作，明显优于行为克隆（Behaviour Cloning）和标准GAIL基线。这项工作揭示了对抗式模仿学习在高维视觉输入场景下失效的根本原因，并提供了一种轻量、通用的修复方案，对推动机器人从像素直接学习人类操作具有重要意义。
原文 arXiv:1910.01077;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.01077v2