Hierarchical Imitation and Reinforcement Learning
Hoang M. Le Nan Jiang Alekh Agarwal Miroslav Dudík Yisong Yue Hal Daumé III
Abstract
We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the hierarchical structure of the underlying problem to integrate different modes of expert interaction. Our framework can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels, leading to dramatic reductions in both expert effort and cost of exploration. Using long-horizon benchmarks, including Montezuma’s Revenge, we demonstrate that our approach can learn significantly faster than hierarchical RL, and be significantly more label-efficient than standard IL. We also theoretically analyze labeling cost for certain instantiations of our framework.
中文速览
稀疏奖励和长时序任务一直是强化学习的老大难问题，现有的模仿学习虽然能借助专家示范缓解这一困境，却往往需要海量标注数据。本文提出了一套名为"层次引导"（hierarchical guidance）的算法框架，将任务天然的两层层次结构（高层选择子目标、低层执行子目标）与模仿学习和强化学习灵活结合：高层专家负责指定子目标序列，低层学习器只在尚未掌握的子任务上接受引导，既避免了对已学会技能的重复标注，又将探索范围收窄到真正相关的状态空间。在著名的高难度游戏《蒙特祖马的复仇》等长时序基准测试中，该方法的学习速度远超纯层次强化学习，所需专家标注量也大幅少于标准模仿学习，同时论文还针对若干具体实例给出了标注代价的理论上界。这项工作的价值在于，它明确了在"专家时间有限"这一现实约束下，如何用最少的人工反馈换取最大的策略学习收益，为构建实用的人机协作学习系统提供了有力支撑。
原文 arXiv:1803.00590;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.00590v2