Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
Benjamin Eysenbach ††\dagger, Shixiang Gu††\dagger ‡‡\ddagger ††††\dagger\dagger, Julian Ibarz††\dagger, Sergey Levine††\dagger ‡‡‡‡\ddagger\ddagger ††\daggerGoogle Brain ‡‡\ddaggerUniversity of Cambridge ††††\dagger\daggerMax Planck Institute for Intelligent Systems ‡‡‡‡\ddagger\ddaggerUC Berkeley Work done as a member of the Google Brain Residency Program (g.co/brainresidency)
Abstract
Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.111Videos of our experiments: https://sites.google.com/site/mlleavenotra
中文速览
让机器人在现实世界中自主学习面临一个棘手问题：每次尝试失败后，都需要人工把环境复原到初始状态，这既费时又费力。本文提出"不留痕迹"（Leave No Trace）框架，让智能体同时学习两个策略——一个"前向策略"负责完成任务，一个"复位策略"负责把环境恢复原状；复位策略的价值函数还能提前预判某个动作是否会把系统带入无法挽回的状态，从而在危险发生前主动"中止"前向策略的行动。实验表明，这套方法大幅减少了人工重置的次数，同时降低了进入不可恢复状态的风险，并自然地为智能体生成了由易到难的学习课程。这项工作为强化学习走向真实机器人场景扫清了一大障碍——让机器人能在更少人工干预下持续、安全地自主学习。
原文 arXiv:1711.06782;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.06782v1