Learning model-based planning from scratch
Razvan Pascanu, Yujia Li∗, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David Reichert,Théophane Weber, Daan Wierstra, Peter Battaglia DeepMind, London, UK {razp, yujiali, vinyals, heess, lbuesing, sracaniere, { reichert, theophane, wierstra, Denotes equal contribution
Abstract
Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to construct a plan. Here we introduce the “Imagination-based Planner”, the first model-based, sequential decision-making agent that can learn to construct, evaluate, and execute plans. Before any action, it can perform a variable number of imagination steps, which involve proposing an imagined action and evaluating it with its model-based imagination. All imagined actions and outcomes are aggregated, iteratively, into a “plan context” which conditions future real and imagined actions. The agent can even decide how to imagine: testing out alternative imagined actions, chaining sequences of actions together, or building a more complex “imagination tree” by navigating flexibly among the previously imagined states using a learned policy. And our agent can learn to plan economically, jointly optimizing for external rewards and computational costs associated with using its imagination. We show that our architecture can learn to solve a challenging contin
中文速览
序贯决策（sequential decision-making）里，基于模型的规划方法一直面临一个核心难题：有了世界模型，你能评估一个计划的好坏，却无法知道该怎么把这个计划构造出来。研究者提出了"基于想象的规划器"（Imagination-based Planner，IBP），让智能体通过反复"想象"来自主学会如何构建、评估并执行计划——每次真实行动前，智能体可以进行任意次数的内部模拟：提出一个假想动作，用世界模型预测其后果，再把所有想象结果积累进一个"计划上下文"来指导后续决策，甚至可以在想象过的状态之间自由跳转、构造出一棵复杂的"想象树"。在连续控制的飞船导航任务和离散的迷宫求解任务中，IBP不仅能学会解决问题，还能自发形成经济高效的规划策略，在完成外部任务的同时压缩计算资源消耗。这项工作首次将规划过程的每一个环节——转移模型、搜索策略、方案汇总——都纳入端到端学习框架，为"学会规划"这一方向提供了新的系统性思路。
原文 arXiv:1707.06170;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.06170v1