Third-Person Imitation Learning
Bradly C. Stadie1,2superscriptBradly C. Stadie12\text{Bradly C. Stadie}^{1,2}, Pieter Abbeel1,3superscriptPieter Abbeel13\text{Pieter Abbeel}^{1,3}, Ilya Sutskever1superscriptIlya Sutskever1\text{Ilya Sutskever}^{1} 1 OpenAI 2 UC Berkeley, Department of Statistics 3 UC Berkeley, Departments of EECS and ICSI {bstadie, pieter,
Abstract
Reinforcement learning (RL) makes it possible to train agents capable of achieving sophisticated goals in complex and uncertain environments. A key difficulty in reinforcement learning is specifying a reward function for the agent to optimize. Traditionally, imitation learning in RL has been used to overcome this problem. Unfortunately, hitherto imitation learning methods tend to require that demonstrations are supplied in the first-person: the agent is provided with a sequence of states and a specification of the actions that it should have taken. While powerful, this kind of imitation learning is limited by the relatively hard problem of collecting first-person demonstrations. Humans address this problem by learning from third-person demonstrations: they observe other humans perform tasks, infer the task, and accomplish the same task themselves.
中文速览
让AI智能体通过"旁观别人干活"来自学技能，是机器人领域的一大难题：传统模仿学习要求给智能体看"第一视角"的示范，但现实中更容易拿到的往往是第三方视角的录像。这篇论文提出了一种无监督的第三视角模仿学习方法，核心思路是借鉴计算机视觉里的"域混淆"（domain confusion）技术，训练一个特征提取器，让它提取出与视角无关的通用表征，再结合生成对抗网络（GAN）框架，用这个表征来区分"专家行为"与"新手行为"，进而给强化学习智能体提供奖励信号。实验在点质量运动、机械臂伸取和倒立摆三个任务上验证了该方法，智能体仅凭不同视角的专家录像、无需任何状态对应标注，就能成功学会完成同样的任务。这项工作迈出了让机器人像人类一样"看视频学技能"的关键一步，为未来低成本、大规模地向机器人传授新技能提供了可行路径。
原文 arXiv:1703.01703;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.01703v2