Latent Offline Model-Based Policy Optimization
Albert Author1 and Bernard D. Researcher2 *This work was not supported by any organization1Albert Author is with Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, 7500 AE Enschede, The Netherlands D. Researcheris with the Department of Electrical Engineering, Wright State University, Dayton, OH 45435, USA
Abstract
Offline reinforcement learning (RL) has shown promise in learning policies from a prerecorded dataset without any interaction with the environment, and has the potential to significantly improve the scalability and safety of policy learning. Recent advances in offline model-based RL have further improved previous offline model-free approaches by enabling greater generalization to new states. These advantages make offline model-based RL particularly appealing for learning complex skills from raw sensor observations, such as images, since learning visuomotor policies using deep RL methods is much more sample-inefficient and unsafe (e.g. failure to generalize to lighting changes) than learning from low-dimensional state inputs. Hence, learning vision-based tasks end-to-end from offline data becomes crucial for robotics and control. However, offline model-based RL from visual inputs is challenging because the offline model-based RL problem critically relies on accurate uncertainty quantification of the model’s predictions to avoid falling off the data distribution and estimating uncertainty of high-dimensional visual dynamics models such as learning an ensemble of video prediction mode
中文速览
离线强化学习（offline RL）通常依赖低维状态输入，而把它扩展到视觉（图像）输入面临两大难题：一是智能体只能用固定数据集训练、不能与环境交互，二是在高维图像空间中量化模型不确定性（如训练一组视频预测模型的集成）既费内存又费算力。为此，研究者提出了 LOMPO（Latent Offline Model-based Policy Optimization），核心思路是先用变分编码器把图像压缩到紧凑的隐空间，再在隐空间中训练一个动力学模型集成，并用集成成员之间的"分歧程度"作为不确定性惩罚项，构造一个保守的惩罚 MDP，最终在这个隐空间惩罚 MDP 中优化策略。理论上可以证明，在隐空间惩罚 MDP 中优化等价于最大化真实 POMDP 回报的一个下界，从而保证了方法的合理性。实验结果显示，LOMPO 在多个图像控制基准（步态运动、机械臂操作及多任务场景，含真实机器人）上显著优于现有的离线无模型方法和在线视觉模型方法，为在真实世界中用纯离线图像数据安全地训练视觉运动策略提供了可行路径。
原文 arXiv:2012.11547;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.11547v1