Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks
Haoqi Yuan1, Chi Zhang2, Hongcheng Wang1,4, Feiyang Xie3, Penglin Cai3, Hao Dong1, Zongqing Lu1,4 1School of Computer Science, Peking University 2School of EECS, Peking University 3Yuanpei College, Peking University 4Beijing Academy of Artificial Intelligence Correspondence to Zongqing Lu Haoqi Yuan
Abstract
We study building multi-task agents in open-world environments. Without human demonstrations, learning to accomplish long-horizon tasks in a large open-world environment with reinforcement learning (RL) is extremely inefficient. To tackle this challenge, we convert the multi-task learning problem into learning basic skills and planning over the skills. Using the popular open-world game Minecraft as the testbed, we propose three types of fine-grained basic skills, and use RL with intrinsic rewards to acquire skills. A novel Finding-skill that performs exploration to find diverse items provides better initialization for other skills, improving the sample efficiency for skill learning. In skill planning, we leverage the prior knowledge in Large Language Models to find the relationships between skills and build a skill graph. When the agent is solving a task, our skill search algorithm walks on the skill graph and generates the proper skill plans for the agent. In experiments, our method accomplishes 40 diverse Minecraft tasks, where many tasks require sequentially executing for more than 10 skills. Our method outperforms baselines by a large margin and is the most sample-efficient dem
中文速览
在《我的世界》这类开放世界游戏中，让智能体不依赖任何人类示范、仅靠强化学习（RL）完成"采矿→制作工具→烹饪食物"这类需要十几步甚至更多步骤的长流程任务，效率极低。为此，研究者提出了Plan4MC框架：将复杂任务分解为三类细粒度基础技能（探索发现、操作交互、合成制作），并引入一个专门的"寻找技能（Finding-skill）"来帮助智能体先在巨大世界中定位目标，从而给其他技能提供良好的起点，大幅提升RL学习效率；在高层规划上，用大语言模型（LLM）预先提取技能间的依赖关系并构建技能图，再由搜索算法在图上动态规划出执行序列，避免直接调用LLM时难以控制的失误。实验在MineDojo模拟器的40个多样化任务上验证，Plan4MC全部完成，且在无需任何示范数据的RL方法中样本效率最高，大幅超越现有基线。这项工作证明了纯RL路线结合层级规划在开放世界多任务智能体中的可行性，为构建通用型游戏/现实世界智能体提供了重要参考。
原文 arXiv:2303.16563;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.16563v2