Useful Policy Invariant Shaping from Arbitrary Advice
Paniz Behboudian , Yash Satsangi , Matthew E. Taylor , Anna Harutyunyan and Michael Bowling
Abstract
Reinforcement learning (RL) is a powerful learning paradigm in which agents can learn to maximize sparse and delayed reward signals. Although RL has had many impressive successes in complex domains, learning can take hours, days, or even years of training data. A major challenge of contemporary RL research is to discover how to learn with less data. Previous work has shown that domain information can be successfully used to shape the reward; by adding additional reward information, the agent can learn with much less data. Furthermore, if the reward is constructed from a potential function, the optimal policy is guaranteed to be unaltered. While such potential-based reward shaping (PBRS) holds promise, it is limited by the need for a well-defined potential function. Ideally, we would like to be able to take arbitrary advice from a human or other agent and improve performance without affecting the optimal policy. The recently introduced dynamic potential based advice (DPBA) method tackles this challenge by admitting arbitrary advice from a human or other agent and improves performance without affecting the optimal policy. The main contribution of this paper is to expose, theoreticall
中文速览
强化学习（Reinforcement Learning）智能体在稀疏奖励环境中学习速度极慢，"奖励塑形"（reward shaping）是一种通过添加额外奖励来加速学习的常见手段，但随意添加奖励会破坏最优策略。一种名为"动态势函数建议"（DPBA）的方法声称能接受任意形式的专家建议、同时保证最优策略不变，然而本文从理论和实验两个层面证明这一核心保证实际上并不成立——DPBA 存在根本性缺陷，会导致智能体收敛到次优策略。为此，作者提出了一种更简单的替代方案"策略不变显式塑形"（PIES），其核心思路是在学习初期让智能体偏向专家建议以快速起步，随后逐渐将这种偏置衰减至零，从而在保证最优策略不变的前提下真正加速学习。这项工作的意义在于纠正了一个已被广泛引用的错误结论，并提供了一个既理论可靠、又实际有效的解决方案。
原文 arXiv:2011.01297;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.01297v1