Learning Latent Dynamics for Planning from Pixels
Danijar Hafner Timothy Lillicrap Ian Fischer Ruben Villegas David Ha Honglak Lee James Davidson
Abstract
Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher tha
中文速览
让智能体在只能看到图像、不知道环境规则的情况下做出高效决策，是强化学习长期面临的核心难题。PlaNet（深度规划网络）的做法是：从原始像素中学习一个紧凑的"世界模型"，该模型同时包含确定性和随机性两条转移路径，能在潜空间（latent space）中快速模拟未来；智能体不需要策略网络或价值网络，而是直接用交叉熵方法在潜空间里搜索最优动作序列，并在每一步重新规划。为了让模型的多步预测更准确，作者还提出了"潜空间超前预测"（latent overshooting）这一新型变分训练目标，让模型在潜空间中同时被所有步长的预测误差监督。在 DeepMind 控制套件的一系列连续控制任务（包含接触动力学、部分可观测和稀疏奖励等挑战）上，PlaNet 平均只需要强模型无关算法约两百分之一的环境交互次数，却能达到相近甚至更高的最终性能，证明了纯基于模型的规划方法在高维视觉输入场景下的巨大潜力。
原文 arXiv:1811.04551;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.04551v5