Pre-Trained Language Models for Interactive Decision-Making
Shuang Li 1 , Xavier Puig1, Chris Paxton2, Yilun Du1, Clinton Wang1, Linxi Fan2, Tao Chen1, De-An Huang2, Ekin Akyürek1, Anima Anandkumar2,3,†, Jacob Andreas1,†, Igor Mordatch4,†, Antonio Torralba1,†, Yuke Zhu2,5,† Correspondence to: Shuang Li 1MIT, 2Nvidia, 3Caltech, 4Google Brain, 5UT Austin Junior authors are ordered based on contributions and senior authors† are ordered alphabetically
Abstract
Language model (LM) pre-training is useful in many language processing tasks. But can pre-trained LMs be further leveraged for more general machine learning problems? We propose an approach for using LMs to scaffold learning and generalization in general sequential decision-making problems. In this approach, goals and observations are represented as a sequence of embeddings, and a policy network initialized with a pre-trained LM predicts the next action. We demonstrate that this framework enables effective combinatorial generalization across different environments and supervisory modalities. We begin by assuming access to a set of expert demonstrations, and show that initializing policies with LMs and fine-tuning them via behavior cloning improves task completion rates by 43.6% in the VirtualHome environment. Next, we integrate an active data gathering procedure in which agents iteratively interact with the environment, relabel past “failed” experiences with new goals, and update their policies in a self-supervised loop. Active data gathering further improves combinatorial generalization, outperforming the best baseline by 25.1%. Finally, we explain these results by investigating t
中文速览
预训练语言模型（LM）向来被视为语言任务的专属工具，但它究竟能不能用来解决与语言无关的序列决策问题？研究者提出了一个叫做 LID 的框架：把智能体的目标、观测和历史动作全部转换成嵌入序列，送入以预训练 GPT-2 初始化的策略网络，再通过行为克隆或主动数据收集进行微调。实验在 VirtualHome 等具身环境中表明，这一做法对"组合泛化"（combinatorial generalization）——即处理训练中从未见过的目标与对象组合——的提升极为显著：相比基线方法，任务完成率最多提高了 43.6%；加入主动数据收集后又额外超越最强基线 25.1%，且无需任何专家示范。进一步分析揭示了关键原因：序列化的输入结构和预训练权重的初始化对泛化都不可或缺，但输入是否采用自然语言字符串却几乎无关紧要，说明语言模型所习得的表征本质上是一种通用的序列建模能力，而非仅对自然语言有效。这一发现为将大规模语言模型迁移到机器人、游戏等非语言决策场景提供了有力的理论依据与实践路径。
原文 arXiv:2202.01771;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.01771v4