Model-Ensemble Trust-Region Policy Optimization
Thanard Kurutach Ignasi Clavera Yan Duan Aviv Tamar Pieter Abbeel Berkeley AI Research University of California, Berkeley Berkeley, CA 94709 {thanard.kurutach, iclavera, rockyduan, avivt,
Abstract
Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and, to date, it has succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and we show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared
中文速览
无模型强化学习（model-free RL）虽然战绩亮眼，却需要海量与环境的交互数据，难以直接用于真实世界；基于模型的强化学习（model-based RL）本可大幅降低数据需求，但当用深度神经网络同时拟合环境模型和策略时，策略优化会主动"钻空子"——专挑训练数据稀少、模型预测失准的区域来刷高虚假回报，导致训练极不稳定。针对这一"模型偏差"问题，论文提出了ME-TRPO方法：用一组初始化和训练顺序各不相同的神经网络集成（ensemble）来表征模型的不确定性，每次想象推演时随机从集成中采样一个模型，迫使策略在多种可能的环境动态下都表现稳健；同时放弃容易引发梯度爆炸或消失的随时间反向传播（BPTT），改用信赖域策略优化（TRPO）这一似然比梯度方法来更新策略，进一步稳定训练过程。实验表明，ME-TRPO在多个具有挑战性的连续控制基准任务上，仅用约百分之一的样本量就达到了当前最优无模型算法的性能水平，为在真实机器人等数据昂贵的场景中应用深度强化学习提供了切实可行的路径。
原文 arXiv:1802.10592;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.10592v2