Can Wikipedia Help Offline Reinforcement Learning?
Machel Reid Yutaro Yamada Shixiang Shane Gu
Abstract
Fine-tuning reinforcement learning (RL) models has been challenging because of a lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models on other domains (vision, language) when finetuned on offline RL tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms of both convergence speed and reward on a variety of environments, accelerating training by 3-6x and achieving state-of-the-art performance in a variety of tasks using Wikipedia-pretrained and GPT2 language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trai
中文速览
用维基百科文章训练过的语言模型，能不能帮助强化学习更快、更好地学会控制机器人和打电子游戏？研究者把离线强化学习（offline RL）重新理解为一种序列建模任务，然后直接把GPT-2这类在自然语言上预训练好的Transformer拿来微调，用于MuJoCo连续控制和Atari游戏这些跟语言毫无关系的任务，同时还引入了两项辅助技术：让输入表示向语言嵌入靠拢的余弦相似度损失，以及联合语言建模的协同训练目标。实验结果显示，相比从零开始训练的Decision Transformer，使用语言预训练模型可以将收敛速度提升3到6倍，同时在多个任务上刷新了最优成绩，平均归一化得分领先基线3到5分以上。这项工作表明，来自完全不同领域的预训练序列模型所学到的通用表示能力可以迁移到强化学习中，为"人人都应在离线RL里用一个预训练语言模型"这一未来图景提供了有力佐证。
原文 arXiv:2201.12122;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.12122v3