How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
Alex X. Lee∗, Coline Devin∗, Jost Tobias Springenberg∗, Yuxiang Zhou, Thomas Lampe, Abbas Abdolmaleki, Konstantinos Bousmalis DeepMind, London, UK ∗ Equal contribution.
Abstract
Reinforcement learning (RL) has been shown to be effective at learning control from experience. However, RL typically requires a large amount of online interaction with the environment. This limits its applicability to real-world settings, such as in robotics, where such interaction is expensive. In this work we investigate ways to minimize online interactions in a target task, by reusing a suboptimal policy we might have access to, for example from training on related prior tasks, or in simulation. To this end, we develop two RL algorithms that can speed up training by using not only the action distributions of teacher policies, but also data collected by such policies on the task at hand. We conduct a thorough experimental study of how to use suboptimal teachers on a challenging robotic manipulation benchmark on vision-based stacking with diverse objects. We compare our methods to offline, online, offline-to-online, and kickstarting RL algorithms. By doing so, we find that training on data from both the teacher and student, enables the best performance for limited data budgets. We examine how to best allocate a limited data budget – on the target task – between the teacher and th
中文速览
真实机器人上的在线强化学习（RL）代价高昂，因为它需要大量与环境的实际交互，而现有的次优教师策略（suboptimal teacher policy）——例如在仿真中或相关任务上训练得到的策略——往往被白白浪费。这项工作系统研究了如何在有限的交互预算下最好地利用这类教师策略，同时结合离线 RL（offline RL）、在线 RL（online RL）与 Kickstarting 三种思路，提出了两种新算法 R-MPO 和 R-CRR，它们既能用教师策略的动作分布作为监督信号，又能直接复用教师在目标任务上采集的轨迹数据。研究者在具有挑战性的视觉机器人堆叠基准 RGB-Stacking 上进行了大规模实验，横跨五种堆叠任务、两种不同次优程度的教师、多种数据预算，并在真实机器人上做了验证。结果表明，混合使用教师数据和学生自身交互数据的方法在数据预算有限时表现最佳，而当数据充足时，纯离线 RL 直接从教师轨迹学习的效果意外地同样具有竞争力——这为实际场景中如何合理分配宝贵的机器人交互资源提供了清晰的实用指导。
原文 arXiv:2205.03353;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.03353v1