Model-Free Episodic Control
Charles Blundell Google DeepMind、Benigno Uria Google DeepMind、Alexander Pritzel Google DeepMind、Yazhe Li Google DeepMind、Avraham Ruderman Google DeepMind、Joel Z Leibo Google DeepMind、Jack Rae Google DeepMind、Daan Wierstra Google DeepMind、Demis Hassabis Google DeepMind
Abstract
State of the art deep reinforcement learning algorithms take many millions of interactions to attain human-level performance. Humans, on the other hand, can very quickly exploit highly rewarding nuances of an environment upon first discovery. In the brain, such rapid learning is thought to depend on the hippocampus and its capacity for episodic memory. Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision-making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.
中文速览
当前最先进的深度强化学习算法需要数千万次与环境的交互才能达到人类水平，而人类一旦发现高回报的策略便能立刻加以利用。受大脑海马体（hippocampus）情景记忆（episodic memory）机制的启发，研究者提出了一种"情景控制"（episodic control）模型：它用一张非参数的近邻查找表记录每个状态下曾经获得的最高回报，遇到相似但未见过的状态时就通过最近邻插值来估计价值，从而实现快速的策略复现。在 Atari 游戏和三维迷宫（Labyrinth）两类环境上的实验表明，该方法不仅比 DQN、A3C 等深度强化学习基线更快地找到高回报策略，在部分难度较高的任务上最终得分也更高。这项工作说明，模拟海马体的快速实例记忆机制可以显著提升强化学习的数据效率，为解决深度强化学习"样本饥渴"这一核心瓶颈提供了新思路。
原文 arXiv:1606.04460;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.04460v1