Language as an Abstraction for Hierarchical Deep Reinforcement Learning
Yiding Jiang , Shixiang Gu, Kevin Murphy, Chelsea Finn Google Research Work done as a part of the Goolge AI Residency program
Abstract
Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi
中文速览
长期以来，让AI智能体完成需要多步骤、长时间规划的复杂任务是强化学习（Reinforcement Learning, RL）领域的难题。这篇论文提出用**自然语言**作为层级强化学习（Hierarchical RL）中高层策略与低层策略之间的"桥梁"：高层策略负责把复杂任务拆解成一条条语言指令，低层策略则逐条执行这些指令来操控物体。为了高效训练低层策略，研究者把"事后经验回放"（Hindsight Experience Replay）推广到了语言指令空间，让智能体能从稀疏奖励中快速学习。他们还基于MuJoCo物理引擎和CLEVR场景生成器构建了一个开源物体交互环境来系统评估方法，结果显示智能体能从像素图像出发，完成物体分类和多物体重新排列等长时序任务，并能泛化到训练时未见过的新指令组合。这项工作的核心发现是：语言的**组合性**（compositionality）——即用有限词汇描述出近乎无限种目标状态——是智能体学会多样子技能并系统性泛化的关键，而使用等量监督但缺乏组合结构的抽象表示则明显逊色，这为构建能理解并执行复杂长程任务的通用AI系统提供了重要启示。
原文 arXiv:1906.07343;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.07343v2