Strategic Attentive Writer for Learning Macro-Actions
Alexander (Sasha) Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu Google DeepMind
Abstract
We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to – i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro
中文速览
强化学习中的智能体通常每看一帧就输出一个动作，既缺乏长远规划，探索效率也低——STRAW（STRategic Attentive Writer）正是为了解决这个问题而提出的。它是一种深度循环神经网络，内部维护两个显式状态：一个"行动计划表"（action-plan），记录未来若干步要采取的动作；一个"承诺计划表"（commitment-plan），决定何时更新计划、何时直接沿用当前计划执行宏观动作（macro-action）。网络借助可微的注意力写入机制（attentive writing）来更新这两张计划表，并通过加入噪声通信通道（类似变分自编码器的做法）实现有结构的随机探索，使得随机性发生在多步计划的层面而非单步动作。实验表明，在需要长期规划的Atari游戏（如Ms. Pacman、Frostbite）上，STRAW相比基线方法取得了显著的分数提升，且在字符预测任务中能自发学会用高频n-gram作为"宏观动作"，证明了方法的通用性。这项工作的意义在于：它完全从奖励信号端到端地学习时间抽象动作，无需人工设定子目标，为强化学习中的层次化规划提供了一条可扩展的新路径。
原文 arXiv:1606.04695;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.04695v1