EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
Aravind Rajeswaran1, Sarvjeet Ghotra2, Balaraman Ravindran3, Sergey Levine4 1 University of Washington Seattle 2 NITK Surathkal 3 Indian Institute of Technology Madras 4 University of California Berkeley
Abstract
Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.
中文速览
真实物理系统上用强化学习训练机器人策略时，既需要海量样本、又面临安全风险，而单一仿真模型与真实环境之间的"仿真-现实差距"又会让纯仿真训练出来的策略一旦迁移就崩溃。EPOpt（集成策略优化，Ensemble Policy Optimization）的思路是同时维护一批参数各异的仿真模型，用条件风险价值（CVaR）作为优化目标，让策略重点在那些表现最差的模型上不断改进——这种对抗式训练迫使策略对各种参数偏差乃至完全未建模的干扰都保持鲁棒；与此同时，每次在真实环境中收集少量数据后，用近似贝叶斯方法更新模型集合的分布，使仿真越来越贴近现实。在MuJoCo的Hopper和Half-Cheetah任务上，EPOpt训练出的策略在直接迁移到真实（或参数偏移的）环境时，显著优于在单一最大似然模型上训练的基线策略，且对未建模效应同样表现出良好泛化性。这项工作为"少踩真实环境、多用仿真、却仍能稳健落地"的机器人学习提供了一条兼顾样本效率与安全性的实用路径。
原文 arXiv:1610.01283;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1610.01283v4