Offline Reinforcement Learning as One Big Sequence Modeling Problem
Michael Janner Qiyang Li Sergey Levine University of California at Berkeley {janner,
Abstract
Reinforcement learning (RL) is typically concerned with estimating stationary policies or single-step models, leveraging the Markov property to factorize problems in time. However, we can also view RL as a generic sequence modeling problem, with the goal being to produce a sequence of actions that leads to a sequence of high rewards. Viewed in this way, it is tempting to consider whether high-capacity sequence prediction models that work well in other domains, such as natural-language processing, can also provide effective solutions to the RL problem. To this end, we explore how RL can be tackled with the tools of sequence modeling, using a Transformer architecture to model distributions over trajectories and repurposing beam search as a planning algorithm. Framing RL as sequence modeling problem simplifies a range of design decisions, allowing us to dispense with many of the components common in offline RL algorithms. We demonstrate the flexibility of this approach across long-horizon dynamics prediction, imitation learning, goal-conditioned RL, and offline RL. Further, we show that this approach can be combined with existing model-free algorithms to yield a state-of-the-art plann
中文速览
强化学习（Reinforcement Learning, RL）长期依赖贝尔曼方程和马尔可夫假设把问题拆解成单步决策，但这篇论文换了一个角度：把整条轨迹（状态、动作、奖励的序列）直接当成自然语言一样的序列来建模，用 Transformer 学习轨迹的联合分布，再把自然语言处理里的集束搜索（Beam Search）改造成规划算法。这一框架把模仿学习、目标导向控制和离线强化学习统一到同一个模型下，省去了单独设计策略网络、动力学模型和行为约束等繁琐组件。实验表明，该"轨迹 Transformer"在离线 RL 基准上与专门设计的最优方法持平，结合动态规划后在稀疏奖励的长时域任务上还达到了当时最优水平。这项工作的意义在于，它证明了大规模序列建模的架构和技术可以几乎原封不动地迁移到强化学习，为借助无监督学习的可扩展性来提升 RL 性能开辟了一条新路。
原文 arXiv:2106.02039;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.02039v4