Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
Felix Hill, Soňa Mokrá, Nathaniel Wong, Tim Harley DeepMind, London
Abstract
Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap
中文速览
用深度强化学习（RL）训练能听懂人类自然语言指令的机器人智能体，长期以来面临一个核心矛盾：RL训练需要海量样本，只能依赖模板句子生成训练指令，而真实用户的表达方式千变万化，导致训练好的智能体根本"听不懂"真人说话。作者提出了一套叫做SHIFTT的方法：先用预训练语言模型BERT赋予智能体强大的语言理解能力，再用模板指令通过RL训练其视觉感知和行为策略，最终让智能体在从未见过人类自然指令的情况下直接接受真人测试。在一个三维仿真房间里的抓取与摆放任务上，实验结果表明，搭载BERT的智能体在"零样本迁移"（zero-shot transfer）条件下执行真人输入指令的准确率远超随机水平，甚至与未经训练的普通人类操作员表现相当。这项工作为所有需要与真实用户对接的深度RL系统提供了一个通用解决方案，也首次将以智能体为中心的运动行为学习与文本预训练表示学习两大研究方向有效桥接起来。
原文 arXiv:2005.09382;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.09382v1