Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution
Vihang Patil Markus Hofmarcher Marius-Constantin Dinu Matthias Dorfer Patrick Blies Johannes Brandstetter José Arjona-Medina Sepp Hochreiter
Abstract
Reinforcement learning algorithms require many samples when solving complex hierarchical tasks with sparse and delayed rewards. For such complex tasks, the recently proposed RUDDER uses reward redistribution to leverage steps in the $Q$ -function that are associated with accomplishing sub-tasks. However, often only few episodes with high rewards are available as demonstrations since current exploration strategies cannot discover them in reasonable time. In this work, we introduce Align-RUDDER, which utilizes a profile model for reward redistribution that is obtained from multiple sequence alignment of demonstrations. Consequently, Align-RUDDER employs reward redistribution effectively and, thereby, drastically improves learning on few demonstrations. Align-RUDDER outperforms competitors on complex artificial tasks with delayed rewards and few demonstrations. On the Minecraft ObtainDiamond task, Align-RUDDER is able to mine a diamond, though not frequently. Code is available at github.com/ml-jku/align-rudder.
中文速览
稀疏奖励与延迟奖励让强化学习（RL）在复杂任务中举步维艰，而现有方法RUDDER虽能通过"奖励再分配"缓解这一问题，却依赖大量高奖励示例来训练其LSTM模型，在示例稀少时效果大打折扣。Align-RUDDER借鉴生物信息学中的多序列比对（Multiple Sequence Alignment，MSA）技术，将少量高奖励示范序列对齐后构建一个位置特异性评分矩阵（PSSM）作为奖励再分配模型，从而只需两三条示例就能有效识别完成子任务的关键事件并分配即时奖励。在多个带延迟奖励的复杂人工任务上，Align-RUDDER的表现显著优于各类基线方法，并在极具挑战性的Minecraft"获取钻石"任务中首次实现了偶发性成功挖钻。这项工作的意义在于，它为示例稀少、探索困难的真实场景提供了一条将生物信息学序列分析与强化学习相融合的新路径，有望推动分层强化学习在复杂长程任务中的实际落地。
原文 arXiv:2009.14108;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.14108v2