My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
Vitaly Kurin Department of Computer Science University of Oxford Oxford, United Kingdom、Maximilian Igl Department of Computer Science University of Oxford Oxford, United Kingdom、Tim Rocktäschel Department of Computer Science University College London London, United Kingdom、Wendelin Böhmer Department of Software Technology Delft University of Technology Delft, Netherlands、Shimon Whiteson Department of Computer Science University of Oxford Oxford, United Kingdom
Abstract
Multitask Reinforcement Learning is a promising way to obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited to compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address incompatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based continuous control uses the physical morphology of the agent to construct the input graph, i.e., encoding limb features as node labels and using edges to connect the nodes if their corresponded limbs are physically connected. In this work, we present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose Amorpheus, a transformer-based approach. Further results show that, while Amorpheus ignores the morphological information that
中文速览
多任务强化学习（Multitask RL）在机器人控制等领域潜力巨大，但当不同任务的状态和动作空间维度不一致时，现有方法普遍依赖图神经网络（GNN）把机器人的物理形态（肢体连接关系）编码进图结构，期望这种"形态先验"能提升策略学习效果。作者通过一系列消融实验发现，把形态信息从图中移除并不会损害GNN方法的性能，说明这一结构偏置并未真正发挥作用；进一步的假设是，稀疏连接图迫使关键信息经过多跳传递，实际上给学习带来了困难，抵消了形态先验的潜在好处。为此，他们提出了基于Transformer的方法Amorpheus——把每个肢体节点视为序列元素，用全连接注意力机制让所有节点直接互相通信，完全绕开多跳传递的问题。实验表明，Amorpheus在样本效率和最终性能上大幅超越依赖形态图结构的GNN基线，这一结果对如何为不兼容多任务控制设计归纳偏置具有重要启示。
原文 arXiv:2010.01856;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.01856v2