Neural Map: Structured Memory for Deep Reinforcement Learning
Emilio Parisotto、Ruslan Salakhutdinov Department of Machine Learning Carnegie Mellon University Pittsburgh, PA 15213, USA
Abstract
A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past $k$ frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past $k$ frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last $k$ frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to env
中文速览
深度强化学习（Deep Reinforcement Learning）智能体在迷宫等部分可观测环境中导航时，需要长期记忆来规划路径，但现有的 LSTM 和记忆网络方案要么容量有限、要么必须靠人工指定存储窗口大小。作者提出了"神经地图"（Neural Map）架构，其核心思想是用一块二维空间结构化的记忆矩阵来模拟现实地图：智能体每走到一个位置，就把当前观测写入对应的地图格子，同时通过全局卷积读取和基于注意力的上下文查询来检索历史信息，从而将空间位置先验知识直接编码进记忆的写入机制中。在一系列 2D 和 3D 迷宫任务上，神经地图的得分显著优于 LSTM 和记忆网络基线，并且能泛化到训练中从未见过的迷宫布局。这项工作表明，把领域知识（"世界是有空间结构的"）注入记忆架构的写操作，可以让智能体以更低的计算代价实现真正意义上的长时程空间推理。
原文 arXiv:1702.08360;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.08360v1