Reward-Free Exploration for Reinforcement Learning
Chi Jin Princeton University Akshay Krishnamurthy Microsoft Research, New York Max Simchowitz University of California, Berkeley Tiancheng Yu Massachusetts Institute of Technology
Abstract
Exploration is widely regarded as one of the most challenging aspects of reinforcement learning (RL), with many naive approaches succumbing to exponential sample complexity. To isolate the challenges of exploration, we propose a new “reward-free RL” framework. In the exploration phase, the agent first collects trajectories from an MDP $\mathcal{M}$ without a pre-specified reward function. After exploration, it is tasked with computing near-optimal policies under for $\mathcal{M}$ for a collection of given reward functions. This framework is particularly suitable when there are many reward functions of interest, or when the reward function is shaped by an external agent to elicit desired behavior.
中文速览
强化学习中的探索问题一直是核心难题——智能体在不知道奖励的情况下，如何高效地了解环境？这篇论文提出了「无奖励强化学习」（reward-free RL）框架：先让智能体在不给定任何奖励函数的情况下自由探索环境、收集数据，之后再针对任意数量的奖励函数各自规划出近似最优策略。作者设计了一个高效算法，只需 Õ(S²A·poly(H)/ε²) 轮探索，就能保证后续对任意奖励函数都能输出 ε 次优策略，并同时证明了近乎匹配的 Ω(S²AH²/ε²) 下界，说明该算法在这一框架下几乎达到最优。这项工作的重要性在于：它把「探索」与「规划」彻底解耦，为需要反复调整奖励函数的实际场景（如约束强化学习、奖励工程）提供了有理论保障的样本高效解决方案，也为后续函数近似等更复杂设置奠定了理论基础。
原文 arXiv:2002.02794;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.02794v1