Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Tianhe Yu∗1, Deirdre Quillen∗2, Zhanpeng He∗3, Ryan Julian∗4, Avnish Narayan∗4, Hayden Shively4, Adithya Bellathur4, Karol Hausman5, Chelsea Finn1, Sergey Levine2 Stanford University1, UC Berkeley2, Columbia University3, University of Southern California4, Robotics at Google5
Abstract
Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while
中文速览
机器人要快速习得新技能，元强化学习（meta-reinforcement learning）是一条很有前景的路，但该领域长期缺乏够"宽"的评测基准——现有测试往往只在跑步速度这类极窄的任务分布上打转，根本无法反映算法是否真能泛化到全新行为。为此，研究者提出了 Meta-World：一个包含 50 种不同桌面机械臂操作任务的开源仿真基准，任务间既有质的区别（如开门、推物、拧瓶盖），又通过随机化物体位置引入参数级变化，兼顾任务多样性与结构共享性。他们在这个基准上系统评测了 7 种当前最先进的元强化学习和多任务强化学习算法，结果令人意外：单个任务尚能学会，但哪怕只同时训练十个不同任务，所有算法的表现就大幅下滑，更遑论泛化到从未见过的新任务。这项工作揭示了现有方法在多任务学习与元学习上的根本局限，所提供的基准和评测协议为推动真正意义上的跨任务泛化研究提供了重要的出发点。
原文 arXiv:1910.10897;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.10897v2