Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
Bradly C. Stadie Department of Statistics University of California, Berkeley Berkeley, CA 94720、Sergey Levine Pieter Abbeel EECS Department University of California, Berkeley Berkeley, CA 94720
Abstract
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space. Hence, exploration in complex domains is often performed with simple epsilon-greedy methods. In this paper, we consider the challenging Atari games domain, which requires processing raw pixel inputs and delayed rewards. We evaluate several more sophisticated exploration strategies, including Thompson sampling and Boltzman exploration, and propose a new exploration method based on assigning exploration bonuses from a concurrently learned model of the system dynamics. By parameterizing our learned model with a neural network, we are able to develop a scalable and efficient approach to exploration bonuses that can be applied to tasks with complex, high-dimensional state spaces. In the Atari domain, our method provides the most consistent improvement across a range of games that pose a major challenge for prior methods. In addition to raw game-scores, we also develop an
中文速览
强化学习中的智能体常常因为探索不足而错过高分区域，在Atari这类需要处理原始像素和延迟奖励的复杂游戏里尤为突出。为了解决这个问题，研究者提出了一种基于"模型预测误差"的探索奖励机制（model-based exploration bonus）：先用自动编码器（autoencoder）把游戏画面压缩成低维特征，再训练一个神经网络去预测下一帧的特征表示，若预测误差大就说明这个状态还很陌生、值得多探索，于是给予额外奖励。与此同时，论文还对Boltzmann探索和Thompson采样等策略进行了横向对比，并提出了AUC-100这一新评估指标，用学习曲线下的面积来衡量探索效率而非单看最终得分。实验结果显示，该方法在Atari多款游戏上带来了最稳定、最显著的提升，尤其在人类玩家远超现有算法的那些"硬探索"游戏上效果突出，说明将可学习的动态模型与探索奖励结合起来是扩展到高维复杂任务的一条可行路径。
原文 arXiv:1507.00814;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1507.00814v3