A Distributional Perspective on Reinforcement Learning
Marc G. Bellemare Will Dabney Rémi Munos
Abstract
In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman’s equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.
中文速览
强化学习通常只建模"期望回报"（Q值），但回报本身是个随机变量，背后藏着完整的概率分布——这篇论文主张应该直接对这个"价值分布"（value distribution）建模，而非仅仅取其均值。作者从理论上证明，在策略评估场景下，分布式贝尔曼算子（distributional Bellman operator）在Wasserstein度量下是一个压缩映射，保证收敛；但在控制场景下，最优性算子并不是任何分布度量下的压缩映射，揭示了一种此前被忽视的不稳定性。基于这些理论洞察，作者设计了一个新算法C51，将贝尔曼方程直接应用于近似值分布的学习，并在Arcade Learning Environment（雅达利游戏基准）上取得了当时的最优成绩。这项工作的重要性在于：它不仅为"为什么建模完整分布比只建模均值更好"提供了理论依据，还通过实验表明完整分布能保留多峰结构、缓解非稳定策略带来的干扰，从而让近似强化学习的训练更加稳定高效。
原文 arXiv:1707.06887;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.06887v1