Temporal Difference Models: Model-Free Deep RL for Model-Based Control
Vitchyr Pong University of California, Berkeley、Shixiang Gu University of Cambridge Max Planck Institute Google Brain \ANDMurtaza Dalal University of California, Berkeley、Sergey Levine University of California, Berkeley denotes equal contribution
Abstract
Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free me
中文速览
强化学习长期被"样本效率低"与"模型偏差大"两难困境所困扰：无模型方法（model-free RL）收敛质量好但需要海量数据，有模型方法（model-based RL）数据高效但因模型误差导致最终性能打折。时序差分模型（Temporal Difference Models, TDM）通过引入一类以目标状态和规划步长为条件的值函数，将两种范式统一在同一框架下——当规划步长为零时，TDM等价于一个隐式动力学模型，可直接用于模型预测控制；随着步长增大，它逐渐过渡到传统无模型的长期预测，从而在保留模型高效利用状态转移信息的同时，避免了模型误差的累积。在多项连续控制任务上的实验表明，TDM的样本效率显著优于主流无模型基线，最终性能也超过了纯有模型方法，真正做到了兼得两者之长。这项工作为弥合有模型与无模型强化学习之间的鸿沟提供了一条清晰的理论路径，对数据获取成本高昂的真实机器人控制等场景具有重要的实际意义。
原文 arXiv:1802.09081;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.09081v2