Combating Reinforcement Learning’s Sisyphean Curse with Intrinsic Fear
Zachary C. Lipton1,2,3 Kamyar Azizzadenesheli4 Abhishek Kumar3 Lihong Li5 Jianfeng Gao6 Li Deng7 Carnegie Mellon University1 Amazon AI2 University of California San Diego3 Univerisity of California Irvine4 Google5 Microsoft Research6 Citadel7 { lihongli jfgao deng } @microsoft.com
Abstract
Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return due to learning on the perturbed objective. We also prove robustness to classification errors. As a bonus, IF models tend to learn faster, owing to reward shaping. Experiments demonstrate that intrinsic-fear DQNs solve otherwise pathological environments and improve on several Atari games.
中文速览
深度强化学习（Deep Reinforcement Learning, DRL）智能体在实际环境中会周期性地"遗忘"那些曾经学过应当避免的灾难性状态，导致它反复犯下同样的致命错误——就像一辆自动驾驶汽车每隔一段时间就得撞几次行人才能"想起"不该撞人。为此，研究者提出了"内在恐惧"（Intrinsic Fear, IF）机制：额外训练一个恐惧模型，专门预测当前状态在未来数步内触发灾难的概率，并将该概率作为惩罚项叠加到Q学习目标上，同时用独立缓冲区永久保存危险状态样本，从根本上防止遗忘。理论分析证明，只要最优策略本身极少进入危险区域，引入这一惩罚对最终回报的损失可以被严格限定在一个极小范围内，且方法对恐惧模型的分类误差具有鲁棒性；实验结果显示，IF-DQN不仅彻底解决了原版DQN在玩具环境中反复失败的问题，还在多款Atari游戏上取得了更高得分。这项工作为在现实世界中安全部署强化学习智能体提供了一条切实可行的路径。
原文 arXiv:1611.01211;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.01211v8