Deep Recurrent Q-Learning for Partially Observable MDPs
Matthew Hausknecht and Peter Stone Department of Computer Science The University of Texas at Austin {mhauskn,
Abstract
Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting Deep Recurrent Q-Network (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN’s performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN’s performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN’s performance degrades less than DQN’s. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN’s input layer and while recurrency confers no systematic advantage when
中文速览
让深度强化学习智能体在信息不完整的环境里也能好好决策，是一个现实中普遍存在却长期被忽视的挑战——传统深度Q网络（DQN）只能看最近四帧画面，一旦关键信息被遮挡就会表现大幅下滑。研究者提出了深度循环Q网络（Deep Recurrent Q-Network，DRQN），做法很简单：把DQN卷积层之后的第一个全连接层换成长短期记忆网络（LSTM），让模型具备跨时间步整合信息的能力。为了验证这一改动的效果，他们设计了"闪烁Pong"任务——每一帧画面有50%概率被完全遮黑，强制要求智能体在残缺观测下仍能推断球的位置和速度——结果DRQN每步只看一帧也能达到与叠帧DQN相当的成绩，且在用完整数据训练后切换到部分观测的评估场景时，性能衰减明显少于DQN。这项工作说明，在观测质量会动态变化或天然存在信息缺失的真实场景中，用循环网络替代帧堆叠是一条切实可行的路径，对将深度强化学习迁移到现实任务具有重要参考价值。
原文 arXiv:1507.06527;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1507.06527v4