Memory-based control with recurrent neural networks
Nicolas Heess* Jonathan J Hunt* Timothy P Lillicrap David Silver Google Deepmind * These authors contributed equally. heess, jjhunt, countzero, davidsilver @ google.com
Abstract
Partially observed control problems are a challenging aspect of reinforcement learning. We extend two related, model-free algorithms for continuous control – deterministic policy gradient and stochastic value gradient – to solve partially observed domains using recurrent neural networks trained with backpropagation through time. We demonstrate that this approach, coupled with long-short term memory is able to solve a variety of physical control problems exhibiting an assortment of memory requirements. These include the short-term integration of information from noisy sensors and the identification of system parameters, as well as long-term memory problems that require preserving information over many time steps. We also demonstrate success on a combined exploration and memory problem in the form of a simplified version of the well-known Morris water maze task. Finally, we show that our approach can deal with high-dimensional observations by learning directly from pixels. We find that recurrent deterministic and stochastic policies are able to learn similarly good solutions to these tasks, including the water maze where the agent must learn effective search strategies.
中文速览
智能体只靠部分观测（看不到完整状态）学会连续控制，是强化学习里的一大难题。研究者把两种无模型的连续控制算法——确定性策略梯度（Deterministic Policy Gradient, DPG）和随机价值梯度（Stochastic Value Gradient, SVG）——都改成用长短期记忆网络（LSTM）作为策略和价值函数的主干，通过时间反向传播让网络自己学会"记什么、忘什么"，从而在不显式维护信念状态的情况下处理部分可观测问题。实验覆盖了多类记忆需求：从融合噪声传感器信息、识别未知系统参数，到需要跨越多步保留信息的长时记忆任务，乃至类似莫里斯水迷宫（Morris water maze）的探索加记忆组合任务，并且还验证了直接从像素学习控制的能力。结果表明，两种循环策略都能有效求解这些任务，且确定性与随机性策略的表现相当接近——这一发现出乎意料，因为通常认为随机策略在部分可观测场景下更有优势。这项工作为在真实世界不完全信息环境中部署深度强化学习提供了一套实用且无需手工设计记忆机制的解决方案。
原文 arXiv:1512.04455;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.04455v1