Noisy Networks for Exploration
Meire Fortunato Mohammad Gheshlaghi Azar Bilal Piot、Jacob Menick Matteo Hessel Ian Osband Alex Graves Volodymyr Mnih、Remi Munos Demis Hassabis Olivier Pietquin Charles Blundell Shane Legg DeepMind {meirefortunato,mazar,piot, jmenick,mtthss,iosband,gravesa,vmnih, Equal contribution.
Abstract
We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent’s policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and Dueling agents (entropy reward and $\epsilon$ -greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.
中文速览
深度强化学习（Deep RL）中的探索问题长期依赖ε-greedy或熵奖励等简单随机扰动策略，这类"局部抖动"方式往往难以驱动智能体产生大规模、有效的探索行为。NoisyNet提出了一种新思路：直接向神经网络的权重中注入参数化噪声，让网络在不同状态下自动学习"该探索多少"，噪声的强度（方差）与其他网络参数一起通过梯度下降来训练，无需人工调节探索超参数。将这一方法分别应用于DQN、Dueling和A3C三种主流算法后，在57款Atari游戏上的测试表明，NoisyNet版本在绝大多数游戏中均大幅超越原始基线，部分游戏中甚至将智能体从低于人类水平直接提升至超越人类水平，且额外计算开销极小。这项工作的意义在于，它提供了一种通用、简洁、可自适应的探索机制，可以即插即用地替换现有深度强化学习算法中的探索启发式方法。
原文 arXiv:1706.10295;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.10295v3