Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
Junhyuk Oh Satinder Singh Honglak Lee Pushmeet Kohli
Abstract
As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.
中文速览
让RL智能体像人类看菜谱做新菜一样、拿着一份从未见过的任务指令清单就能完成复杂序列任务——这就是这篇论文要攻克的核心难题。作者将问题拆成两个阶段：先用"类比学习"目标函数（analogy-making objective）训练一个参数化技能网络，让智能体通过学习不同子任务之间的对应关系，在嵌入空间里构建出平行四边形结构，从而推广到训练时从未见过的子任务组合；再在此基础上搭建一个层级式元控制器（hierarchical meta-controller），配备一个专门学习"何时切换子任务"的新型神经网络模块，使智能体在只拿到全部指令、没有中间反馈的情况下，仍能自主判断当前指令何时完成、何时插入处理突发事件。在基于Minecraft搭建的随机3D环境中，实验证明类比目标对泛化到未见指令至关重要，而元控制器的子任务切换机制则是顺利执行更长指令序列的关键，两者缺一不可，并显著优于多个层级基线方法。这项工作的价值在于，它为RL智能体实现"零样本任务泛化"——无需重新训练便能应对新指令——提供了一套可落地的方法论框架。
原文 arXiv:1706.05064;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.05064v2