Online Decision Transformer
Qinqing Zheng Amy Zhang Aditya Grover
Abstract
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via task-specific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning. Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure.
中文速览
强化学习（RL）智能体在离线数据集上预训练后性能往往受限，还需要通过与环境的在线交互来进一步提升，但如何把"离线预训练"和"在线微调"无缝衔接起来，一直是个难题。这篇论文提出了"在线决策变换器"（Online Decision Transformer，ODT），在决策变换器（Decision Transformer）的序列建模框架基础上，引入了序列级别的最大熵正则化目标，让模型在在线探索阶段能主动寻找新经验而不是保守地重复已知行为；同时设计了与架构配套的轨迹回放缓冲区，并借鉴"后见经验回放"思想对采集到的轨迹进行回报标签修正，从而让在线数据能被高效利用。实验在D4RL标准基准上验证，ODT在绝对性能上与最先进方法持平，而在微调阶段的提升幅度则明显优于现有基线。这项工作的意义在于，它证明了以语言模型为原型的序列建模范式不仅能做离线RL，也能优雅地扩展到需要主动探索的在线场景，为大规模预训练与任务特定微调的统一框架提供了可行路径。
原文 arXiv:2202.05607;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.05607v2