Learning to Reach Goals via Iterated Supervised Learning
Dibya Ghosh UC Berkeley、Abhishek Gupta∗ UC Berkeley、Ashwin Reddy UC Berkeley、Justin Fu UC Berkeley Coline Devin UC Berkeley、Benjamin Eysenbach Carnegie Mellon University、Sergey Levine UC Berkeley First two authors contributed equally. Correspondence at
Abstract
Current reinforcement learning (RL) algorithms can be brittle and difficult to use, especially when learning goal-reaching behaviors from sparse rewards. Although supervised imitation learning provides a simple and stable alternative, it requires access to demonstrations from a human supervisor. In this paper, we study RL algorithms that use imitation learning to acquire goal reaching policies from scratch, without the need for expert demonstrations or a value function. In lieu of demonstrations, we leverage the property that any trajectory is a successful demonstration for reaching the final state in that same trajectory. We propose a simple algorithm in which an agent continually relabels and imitates the trajectories it generates to progressively learn goal-reaching behaviors from scratch. Each iteration, the agent collects new trajectories using the latest policy, and maximizes the likelihood of the actions along these trajectories under the goal that was actually reached, so as to improve the policy. We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstra
中文速览
训练机器人从零学会"走到指定目标位置"是强化学习的经典难题，稀疏奖励信号让算法又慢又不稳定。这篇论文提出了一个叫做目标条件监督学习（Goal-Conditioned Supervised Learning，GCSL）的方法：核心思路是把智能体自己走出来的每条轨迹"重新打标签"——虽然这条路没走到原定目标，但它确实走到了某个终点，那它对那个终点来说就是一条完美的示范；用这些"事后打标签"的轨迹反复做行为克隆（behavioral cloning），智能体就能在不依赖任何人工示范、也不需要学习值函数的前提下，一步步改进自己的目标导向策略。论文从理论上证明了这个迭代过程等价于优化强化学习目标函数的一个下界，并推导出策略性能的收敛保证；在多个机器人控制基准任务的实验中，GCSL比主流的基于值函数的强化学习方法更稳定、对超参数更不敏感，整体表现也更好。这项工作的意义在于，它用最简单的监督学习框架就实现了从头学习目标导向行为，大幅降低了强化学习在实际部署中的调参门槛。
原文 arXiv:1912.06088;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.06088v4