HIGhER : Improving instruction following with Hindsight Generation for Experience Replay
Geoffrey Cideron Université de Lille CRIStAL, CNRS, Inria France Mathieu Seurin Université de Lille CRIStAL, CNRS, Inria France Florian Strub DeepMind Paris France Olivier Pietquin Google Research Brain Team, Paris France
Abstract
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it comple
中文速览
用语言指令来训练智能体（agent）做任务时，奖励信号极其稀疏——只有完成任务才有正反馈，大量失败经历都被白白浪费了。作者提出了 HIGhER（Hindsight Generation for Experience Replay，后见生成式经验回放），核心思路是：每当智能体没完成指令时，用一个额外训练的"指令生成器"根据智能体实际走过的轨迹反推出一条与之匹配的新指令，再用这条新指令把失败轨迹重新标记为成功，从而凭空制造出更多有正奖励的训练样本。这个指令生成器本身只靠环境里自然产生的成功轨迹来监督学习，不需要任何人工标注或外部专家介入。在 BabyAI 视觉导航环境中的实验表明，HIGhER 显著优于纯强化学习基线，而且对生成指令的噪声具有较强鲁棒性，还能与其他指令跟随方法互补叠加使用，为在语言条件下解决稀疏奖励问题提供了一条无需人工干预的通用路径。
原文 arXiv:1910.09451;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.09451v3