Generalization and Regularization in DQN
Jesse Farebrother∗1, Marlos C. Machado2, and Michael Bowling1,3
Abstract
Deep reinforcement learning algorithms have shown an impressive ability to learn complex control policies in high-dimensional tasks. However, despite the ever-increasing performance on popular benchmarks, policies learned by deep reinforcement learning algorithms can struggle to generalize when evaluated in remarkably similar environments. In this paper we propose a protocol to evaluate generalization in reinforcement learning through different modes of Atari 2600 games. With that protocol we assess the generalization capabilities of DQN, one of the most traditional deep reinforcement learning algorithms, and we provide evidence suggesting that DQN overspecializes to the training environment. We then comprehensively evaluate the impact of dropout and $\ell_{2}$ regularization, as well as the impact of reusing learned representations to improve the generalization capabilities of DQN. Despite regularization being largely underutilized in deep reinforcement learning, we show that it can, in fact, help DQN learn more general features. These features can be reused and fine-tuned on similar tasks, considerably improving DQN’s sample efficiency.
中文速览
深度强化学习（Deep Reinforcement Learning）算法虽然在各类基准测试上屡创佳绩，但训练好的智能体换一个哪怕只有细微差异的环境往往就"认不出来"了。研究者利用雅达利2600游戏（Atari 2600）内置的不同模式与难度级别——这些"口味"改变了游戏画面、物体速度等表观特征，但保留了核心玩法——搭建了一套评估泛化能力的实验框架，并用它系统测试了经典算法DQN，结果发现DQN明显存在对训练环境过拟合（overfitting）的倾向。进一步实验表明，将监督学习中常用却在强化学习里鲜少使用的dropout和ℓ₂正则化引入DQN，可以让它学到更通用的特征表示，当遇到新任务时，这些表示能够被直接迁移并微调（fine-tuning），显著提升样本效率。这项工作不仅为强化学习的泛化评估提供了第一批基准结果，也揭示了正则化在值函数学习中被长期忽视的潜力，对开发真正能举一反三的智能体具有重要参考价值。
原文 arXiv:1810.00123;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.00123v3