Towards Generalization and Simplicity in Continuous Control
Aravind Rajeswaran∗ Kendall Lowrey∗ Emanuel Todorov Sham Kakade University of Washington Seattle {{\{ aravraj, klowrey, todorov, sham }}\} @ cs.washington.edu
Abstract
This work shows that policies with simple linear and RBF parameterizations can be trained to solve a variety of widely studied continuous control tasks, including the OpenAI gym benchmarks. The performance of these trained policies are competitive with state of the art results, obtained with more elaborate parameterizations such as fully connected neural networks. Furthermore, the standard training and testing scenarios for these tasks are shown to be very limited and prone to over-fitting, thus giving rise to only trajectory-centric policies. Training with a diverse initial state distribution induces more global policies with better generalization. This allows for interactive control scenarios where the system recovers from large on-line perturbations; as shown in the supplementary video.
中文速览
深度强化学习在连续控制任务上的成功长期被归因于多层神经网络的强大表达能力，但这一假设从未被严格验证。研究者用线性策略和RBF（径向基函数）随机特征策略，配合自然策略梯度训练，在OpenAI Gym的MuJoCo连续控制基准任务上进行了对比实验，结果发现这两种极简策略的表现与深度神经网络相当甚至更优，且线性策略的训练速度快近20倍。更重要的是，研究揭示了现有基准任务存在严重的过拟合问题——由于初始状态分布过于狭窄，学到的策略只在训练时常见的轨迹附近有效，稍加扰动便彻底失败；而当采用更多样化的初始状态分布进行训练后，策略的鲁棒性显著提升，能够从跌倒等大幅扰动中自主恢复，且在这一更严格的设定下简单策略依然不逊于神经网络。这项工作提醒研究者：当前流行的基准任务不足以说明复杂网络架构的必要性，在追求更复杂方法之前，应当先把简单基线和任务设计本身搞清楚。
原文 arXiv:1703.02660;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.02660v2