Large Language Models Are Semi-Parametric Reinforcement Learning Agents
Danyang Zhang1、Lu Chen1,212{}^{1,\>\!2} 、Situo Zhang1 Hongshen Xu1、Zihan Zhao1、Kai Yu1,212{}^{1,\>\!2} Lu Chen is the corresponding author. 1X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China 2Suzhou Laboratory, Suzhou, China
Abstract
Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as Rememberer. By equipping the LLM with a long-term experience memory, Rememberer is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed Rememberer constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of Rememberer.111The codes are open-sourced at https://github.com/OpenDFM/Rememberer.
中文速览
让大型语言模型（LLM）真正从过去的交互经验中学习、持续进化，是当前智能体研究的核心难题——现有方法要么只能用固定示例，要么只有一次性的短期记忆，无法跨任务目标积累经验。论文提出了一个名为 Rememberer 的智能体框架，为 LLM 配备一个外部持久化的长期经验记忆库，并设计了"基于经验记忆的强化学习"（RLEM，Reinforcement Learning with Experience Memory）机制，通过类 Q-Learning 的方式持续更新记忆中每条经验的价值估计，让智能体既能从成功中借鉴、也能从失败中规避，而全程无需微调 LLM 参数。决策时，智能体根据当前状态检索最相关的历史经验作为动态少样本示例，引导 LLM 做出更优决策，形成一个"半参数化强化学习智能体"。在 WebShop 和 WikiHow 两个基准上的实验表明，Rememberer 的成功率分别比此前最优方法提升了 2 个和 4 个百分点，证明跨任务的长期经验积累能够有效且持续地增强 LLM 智能体的决策能力。
原文 arXiv:2306.07929;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.07929v2