Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
Abstract
We propose a “plan online and learn offline” framework for the setting where an agent, with an internal model, needs to continually act and learn in the world. Our work builds on the synergistic relationship between local model-based control, global value function learning, and exploration. We study how local trajectory optimization can cope with approximation errors in the value function, and can stabilize and accelerate value function learning. Conversely, we also study how approximate value functions can help reduce the planning horizon and allow for better policies beyond local solutions. Finally, we also demonstrate how trajectory optimization can be used to perform temporally coordinated exploration in conjunction with estimating uncertainty in value function approximation. This exploration is critical for fast and stable learning of the value function. Combining these components enable solutions to complex control tasks, like humanoid locomotion and dexterous in-hand manipulation, in the equivalent of a few minutes of experience in the real world.
中文速览
机器人在真实世界中边行动边学习，既要实时做决策又要不断积累经验，单靠轨迹优化（trajectory optimization）容易陷入局部最优，单靠值函数（value function）近似又会因误差随时间步累积而导致策略变差。作者提出了"在线规划、离线学习"框架POLO，将局部模型预测控制（MPC）、全局值函数近似和有目的的探索三者紧密结合：MPC利用近似值函数缩短规划视野并抑制误差放大，近似值函数则借助MPC生成的多步Bellman目标加速收敛，二者共同支撑一种"时序协调探索"——智能体对潜在高奖励区域形成假设后，用MPC规划出整条探索轨迹，而非逐步随机试探。实验表明，在仿人机器人行走和灵巧手指内操作等复杂连续控制任务上，POLO只需相当于真实世界几分钟的交互经验便能习得高质量策略，为样本效率极低的复杂机器人控制任务提供了一条可行路径。
原文 arXiv:1811.01848;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.01848v3