Offline Learning from Demonstrations and Unlabeled Experience
Konrad Żołna1、Alexander Novikov1、Ksenia Konyushkova1、Caglar Gulcehre1、Ziyu Wang2、Yusuf Aytar1、Misha Denil1、Nando de Freitas1、Scott Reed1 1DeepMind 2Google Brain
Abstract
Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. However, BC does not effectively leverage what we will refer to as unlabeled experience: data of mixed and unknown quality without reward annotations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. Towards data-driven offline robot learning that can use this unlabeled experience, we introduce Offline Reinforced Imitation Learning (ORIL). ORIL first learns a reward function by contrasting observations from demonstrator and unlabeled trajectories, then annotates all data with the learned reward, and finally trains an agent via offline reinforcement learning. Across a diverse set of continuous control and simulated robotic manipulation tasks, we show that ORIL consistently outperforms comparable BC agents by effectively leveraging unlabeled experience.
中文速览
行为克隆（Behavior Cloning）在机器人学习中很常用，但它只能利用专家示范数据，无法有效使用大量来源混杂、质量参差不齐、也没有奖励标注的"无标签经验"。为此，研究者提出了离线强化模仿学习方法 ORIL（Offline Reinforced Imitation Learning）：先通过对比专家轨迹与无标签轨迹来训练一个奖励模型，再用这个模型为所有数据自动打上奖励分数，最后在完整数据集上运行离线强化学习算法来训练策略。在模拟机械臂操作和 DeepMind Control Suite 等多类连续控制任务上，ORIL 的表现比同类行为克隆基线高出 20%–100%，甚至接近使用真实奖励的上界，而且无标签数据越多、性能越好。这项工作的意义在于，它为机器人提供了一条无需人工标注奖励、同时又能充分榨取海量历史交互数据价值的实用学习路径。
原文 arXiv:2011.13885;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.13885v1