How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
Vincent Fran ois-Lavet University of Liege、Raphael Fonteneau University of Liege \ANDDamien Ernst University of Liege
Abstract
Using deep neural nets as function approximator for reinforcement learning tasks have recently been shown to be very powerful for solving problems approaching real-world complexity such as [1]. Using these results as a benchmark, we discuss the role that the discount factor may play in the quality of the learning process of a deep Q-network (DQN). When the discount factor progressively increases up to its final value, we empirically show that it is possible to significantly reduce the number of learning steps. When used in conjunction with a varying learning rate, we empirically show that it outperforms original DQN on several experiments. We relate this phenomenon with the instabilities of neural networks when they are used in an approximate Dynamic Programming setting. We also describe the possibility to fall within a local optimum during the learning process, thus connecting our discussion with the exploration/exploitation dilemma.
中文速览
训练深度强化学习智能体时，折扣因子（discount factor）的设置会显著影响学习稳定性和收敛速度——设得太高会导致神经网络产生误差累积甚至发散，而一开始就固定在高值正是常见做法的潜在隐患。受儿童"延迟满足"实验启发，作者提出让折扣因子从较小值逐步增大到最终目标值，同步配合学习率从大到小衰减，使网络在早期低折扣阶段用更激进的步长快速学习短期策略，再随折扣提升而趋于稳定。在经典Atari游戏基准上的实验表明，这种渐进式调整策略在五款游戏中有四款实现了更快的策略提升，结合自适应学习率后在全部六款测试游戏上均超越了原版DQN的得分，且对网络权重初始化更加鲁棒。这项工作揭示了折扣因子在深度Q学习中兼具"正则化"与"稳定训练"的双重作用，为后续将自适应折扣、探索策略与并行/对抗网络等前沿方法结合提供了清晰的研究方向。
原文 arXiv:1512.02011;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.02011v2