Learning to Compose Skills
Himanshu Sahni Georgia Institute of Technology \AndSaurabh Kumar Georgia Institute of Technology \ANDFarhan Tejani Georgia Institute of Technology \AndCharles Isbell Georgia Institute of Technology Denotes equal contribution.
Abstract
We present a differentiable framework capable of learning a wide variety of compositions of simple policies that we call skills. By recursively composing skills with themselves, we can create hierarchies that display complex behavior. Skill networks are trained to generate skill-state embeddings that are provided as inputs to a trainable composition function, which in turn outputs a policy for the overall task. Our experiments on an environment consisting of multiple collect and evade tasks show that this architecture is able to quickly build complex skills from simpler ones. Furthermore, the learned composition function displays some transfer to unseen combinations of skills, allowing for zero-shot generalizations.
中文速览
强化学习智能体往往只会从零开始学一项复杂任务，很难把已掌握的简单技能灵活组合起来复用。这篇论文提出了一个叫做 ComposeNet 的可微组合框架：先分别训练若干基础技能网络，让它们各自生成一个"技能-状态嵌入"向量来描述当前状态与技能目标的关系，再用一个可训练的组合层把多个嵌入融合成复合任务的嵌入，最终经由共享策略层输出动作。在一个需要收集或规避多种彩色物体的类 Pacman 环境中，实验表明 ComposeNet 组合已有技能的速度远快于从头学习复杂任务，而且组合层还能迁移到从未见过的技能组合，实现零样本泛化（zero-shot generalization）。这项工作的意义在于，它提供了一条把简单技能像搭积木一样递归组合成复杂行为的高效路径，让技能真正做到"一次学习、反复复用"。
原文 arXiv:1711.11289;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.11289v1