Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
Caglar Gulcehre Equal contributions DeepMind, London Tom Le Paine Equal contributions DeepMind, London Bobak Shahriari DeepMind, London Misha Denil DeepMind, London Matt Hoffman DeepMind, London Hubert Soyer DeepMind, London Richard Tanburn DeepMind, London Steven Kapturowski DeepMind, London Neil Rabinowitz DeepMind, London Duncan Williams DeepMind, London Gabriel Barth-Maron DeepMind, London Ziyu Wang DeepMind, London Nando de Freitas DeepMind, London Worlds Team DeepMind, London
Abstract
This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.
中文速览
强化学习在稀疏奖励环境中的探索难题长期困扰研究者，而当环境同时具备部分可观测性和每局随机初始条件时，问题更是难上加难——示范数据中的动作序列无法直接复用，智能体必须真正"理解"任务而非死记硬背。R2D3（Recurrent Replay Distributed DQN from Demonstrations）将专家示范与分布式循环Q学习相结合，设计了两个独立的优先级经验回放池分别存储专家轨迹和智能体自身经验，并通过一个称为"示范比例"（demo ratio）的关键超参数来控制二者在训练批次中的混合比例——实验发现最优比例极小但不为零，其作用在于将示范数据作为引导探索的"指南针"，而非直接模仿的脚本。为了系统评估这一方法，作者还设计了包含八个3D程序化生成任务的Hard-Eight测试集，要求智能体完成工具使用、长程记忆等复杂行为。结果表明，R2D3在多个任务上成功找到解法，而其他最先进方法（包括有无示范的各类基线）在数百亿步探索后仍未见到哪怕一条成功轨迹，这项工作为在真实多变场景中高效利用少量示范数据提供了重要思路。
原文 arXiv:1909.01387;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.01387v1