Universal Planning Networks
Aravind Srinivas Allan Jabri Pieter Abbeel Sergey Levine Chelsea Finn
Abstract
A key challenge in complex visuomotor control is learning abstract representations that are effective for specifying goals, planning, and generalization. To this end, we introduce universal planning networks (UPN). UPNs embed differentiable planning within a goal-directed policy. This planning computation unrolls a forward model in a latent space and infers an optimal action plan through gradient descent trajectory optimization. The plan-by-gradient-descent process and its underlying representations are learned end-to-end to directly optimize a supervised imitation learning objective. We find that the representations learned are not only effective for goal-directed visual imitation via gradient-based trajectory optimization, but can also provide a metric for specifying goals using images. The learned representations can be leveraged to specify distance-based rewards to reach new target states for model-free reinforcement learning, resulting in substantially more effective learning when solving new tasks described via image-based goals. We were able to achieve successful transfer of visuomotor planning strategies across robots with significantly different morphologies and actuation
中文速览
视觉运动控制中，让机器根据一张目标图片学会规划并执行动作，难点在于如何自动获得一个"可供规划使用"的抽象状态表征。通用规划网络（Universal Planning Networks, UPN）将一个可微分的梯度下降规划器嵌入策略网络内部，让模型在学到的隐空间里模拟未来轨迹、通过反向传播优化动作序列，并以模仿学习为外层目标端到端地训练整个系统，从而直接逼迫网络学出"适合梯度规划"的表征。实验表明，这样学到的隐空间不仅支持高效的视觉模仿，还天然构成一个有意义的距离度量，可作为奖励信号直接用于无模型强化学习（model-free RL）来完成新任务，显著优于基于原始像素误差的奖励设计。更重要的是，这套习得的规划策略能够跨形态差异显著的不同机器人迁移，为复杂视觉运动任务的泛化与迁移提供了一条无需人工设计奖励函数的新路径。
原文 arXiv:1804.00645;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.00645v2