Generalized Decision Transformer for Offline Hindsight Information Matching
Hiroki Furuta The University of Tokyo、Yutaka Matsuo The University of Tokyo、Shixiang Shane Gu Google Research
Abstract
How to extract as much learning signal from each trajectory data has been a key problem in reinforcement learning (RL), where sample inefficiency has posed serious challenges for practical applications. Recent works have shown that using expressive policy function approximators and conditioning on future trajectory information – such as future states in hindsight experience replay (HER) or returns-to-go in Decision Transformer (DT) – enables efficient learning of multi-task policies, where at times online RL is fully replaced by offline behavioral cloning (BC), e.g. sequence modeling. We demonstrate that all these approaches are doing hindsight information matching (HIM) – training policies that can output the rest of trajectory that matches some statistics of future state information. We present Generalized Decision Transformer (GDT) for solving any HIM problem, and show how different choices for the feature function and the anti-causal aggregator not only recover DT as a special case, but also lead to novel Categorical DT (CDT) and Bi-directional DT (BDT) for matching different statistics of the future. For evaluating CDT and BDT, we define offline multi-task state-marginal match
中文速览
强化学习长期面临样本利用率低的痛点，而"后见之明"类算法（hindsight algorithms）通过把轨迹未来信息喂给策略来压榨每条数据的学习信号，涵盖了从稀疏奖励Q学习（HER）到用Transformer做离线强化学习（Decision Transformer, DT）等一大批方法。本文提出"后见信息匹配"（Hindsight Information Matching, HIM）这一统一框架，揭示上述看似各异的算法本质上都在训练策略去复现轨迹未来状态的某种统计量，并在此基础上推出广义决策Transformer（Generalized Decision Transformer, GDT），只需改变特征函数和反因果聚合器这两个模块，就能派生出针对不同统计目标的新变体：用"反因果分箱"替换求和得到Categorical DT（CDT），用双向Transformer做聚合器得到Bi-directional DT（BDT）。实验在MuJoCo连续控制任务上表明，CDT是首个有效的离线多任务状态边际匹配算法，能泛化到未见过的多峰分布；BDT则在离线多任务模仿学习（单样本模仿）上超越了现有DT变体。这项工作大幅拓展了序列建模架构在强化学习中的应用边界，为后续结合大规模Transformer解决更广泛RL问题提供了系统性的理论基础。
原文 arXiv:2111.10364;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.10364v3