Prioritized Experience Replay
Tom Schaul, John Quan, Ioannis Antonoglou and David Silver Google DeepMind
Abstract
Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.
中文速览
强化学习智能体在训练时通常把经历过的"经验片段"存入记忆池并均匀随机抽取回放，但这样做忽视了不同经验的学习价值差异。本文提出"优先经验回放"（Prioritized Experience Replay）框架，核心思路是用TD误差的大小来衡量每条经验的学习价值，让智能体更频繁地回放那些"意外性"更高、更值得学习的经验；同时引入随机优先采样避免过拟合，并用重要性采样权重修正由此带来的分布偏差。将这一方法整合进DQN后，在49款Atari游戏中的41款上超越了原始均匀回放的DQN，取得了新的最优成绩，同时学习速度也显著加快。这项工作意义在于：在不增加与环境交互次数的前提下，仅通过更聪明地复用已有经验就大幅提升了深度强化学习的样本效率，为后续优先回放研究奠定了基础。
原文 arXiv:1511.05952;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.05952v4