Model-Based Reinforcement Learning via Meta-Policy Optimization
Ignasi Clavera UC Berkeley、Jonas Rothfuss∗ KIT, UC Berkeley、John Schulman OpenAI、Yasuhiro Fujita Preferred Networks、Tamim Asfour Karlsruhe Inst. of Technology (KIT)、Pieter Abbeel UC Berkeley, Covariant.AI Equal contribution
Abstract
Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.
中文速览
基于模型的强化学习（Model-Based RL）虽然数据效率高，但学出来的动力学模型往往不够精准，导致策略性能远不如无模型（Model-Free）方法——这个"模型偏差"问题长期制约着该领域的发展。MB-MPO 的思路是：训练一组动力学模型（ensemble），再用元学习框架 MAML 让策略"学会快速适应"这组模型中的任意一个，只需一步策略梯度即可完成适应。这样一来，元策略自然会吸收各模型一致认同的动力学规律，而对模型之间的分歧则留给适应步骤来处理，从根本上降低了对单一精确模型的依赖。实验表明，MB-MPO 在多个复杂连续控制任务上不仅超越了以往所有基于模型的方法，而且能够匹敌无模型方法的最终性能，同时所需的真实环境交互数据量少得多——例如在四足机器人运动任务中，两小时的真实数据就足够了，而无模型方法通常需要多出10到100倍的数据。
原文 arXiv:1809.05214;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.05214v1