Model-Based Value Expansion for Efficient Model-Free Reinforcement Learning
Vladimir Feinberg Alvin Wan Ion Stoica Michael I. Jordan Joseph E. Gonzalez Sergey Levine
Abstract
Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.
中文速览
真实环境中收集数据代价高昂，如何让智能体用更少的环境交互就学会连续控制任务，是强化学习落地的核心瓶颈。这篇论文提出了"基于模型的价值展开"（Model-Based Value Expansion，MVE）方法：在无模型强化学习的Q值估计阶段，借用一个学到的环境动力学模型向前"想象"固定步数，用这段想象轨迹的真实奖励加上末端的无模型价值估计来构造更准确的学习目标，超出模型可信深度的部分则交还给无模型方法处理。实验表明，与纯无模型算法及以往混合方法相比，MVE能显著降低样本复杂度，在连续控制基准任务上用更少的真实环境交互达到相同甚至更好的性能。这项工作的价值在于，它给出了一个有理论保证、无需可微动力学假设的模型利用方式，并通过"TD-k技巧"解决了模型生成数据与真实数据分布不匹配这一此前被忽视的关键问题，为将动力学模型更系统地嵌入无模型框架提供了可靠路径。
原文 arXiv:1803.00101;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.00101v1