DeepMDP: Learning Continuous Latent Space Models for Representation Learning
Carles Gelada Saurabh Kumar Jacob Buckman Ofir Nachum Marc G. Bellemare
Abstract
Many reinforcement learning (RL) tasks provide the agent with high-dimensional observations that can be simplified into low-dimensional continuous states. To formalize this process, we introduce the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: prediction of rewards and prediction of the distribution over next latent states. We show that the optimization of these objectives guarantees (1) the quality of the latent space as a representation of the state space and (2) the quality of the DeepMDP as a model of the environment. We connect these results to prior work in the bisimulation literature, and explore the use of a variety of metrics. Our theoretical findings are substantiated by the experimental result that a trained DeepMDP recovers the latent structure underlying high-dimensional observations on a synthetic environment. Finally, we show that learning a DeepMDP as an auxiliary task in the Atari 2600 domain leads to large performance improvements over model-free RL.
中文速览
强化学习（Reinforcement Learning, RL）智能体常常面对高维图像输入（如Atari游戏的33600维像素），但真正有用的信息其实藏在低维的潜在结构里。作者提出了"DeepMDP"框架，通过训练两个易于优化的损失函数——预测奖励和预测下一潜在状态的分布——来学习一个压缩的潜在空间模型（latent space model）。理论上，他们证明了只要把这两个损失最小化，潜在空间中的值函数就能可靠地逼近原始MDP中的值函数，并揭示了DeepMDP与经典双仿真度量（bisimulation metric）之间的深刻联系。实验结果表明，DeepMDP能在合成环境中精准还原高维观测背后的低维结构，并且当把它作为辅助任务加入Atari 2600的无模型RL训练时，性能获得了大幅提升。这项工作的意义在于，它首次把深度学习友好的潜在模型与严格的理论保证结合起来，为可扩展的、有原则的表示学习提供了坚实基础。
原文 arXiv:1906.02736;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.02736v1