ExpeL: LLM Agents Are Experiential Learners
Andrew Zhao,♠♠\spadesuit♠ Daniel Huang, ♣♣\clubsuit♣ Quentin Xu, ♣♣\clubsuit♣ Matthieu Lin, ♣♣\clubsuit♣ Yong-Jin Liu, ♣♣\clubsuit♣ Gao Huang ♠♠\spadesuit♠ Corresponding author.
Abstract
The recent surge in research interest in applying large language models (LLMs) to decision-making tasks has flourished by leveraging the extensive world knowledge embedded in LLMs. While there is a growing demand to tailor LLMs for custom decision-making tasks, finetuning them for specific tasks is resource-intensive and may diminish the model’s generalization capabilities. Moreover, state-of-the-art language models like GPT-4 and Claude are primarily accessible through API calls, with their parametric weights remaining proprietary and unavailable to the public. This scenario emphasizes the growing need for new methodologies that allow learning from agent experiences without requiring parametric updates. To address these problems, we introduce the Experiential Learning (ExpeL) agent. Our agent autonomously gathers experiences and extracts knowledge using natural language from a collection of training tasks. At inference, the agent recalls its extracted insights and past experiences to make informed decisions. Our empirical results highlight the robust learning efficacy of the ExpeL agent, indicating a consistent enhancement in its performance as it accumulates experiences. We furth
中文速览
大规模语言模型（LLM）在决策任务中展现出巨大潜力，但现有方法要么需要昂贵的参数微调、要么只能在单次任务内反思改进，无法跨任务积累经验。为此，研究者提出了"经验学习智能体"ExpeL（Experiential Learning Agent）：在训练阶段，它通过反复试错自主收集任务轨迹，再用语言模型从成功与失败的对比中提炼出自然语言规则（insights），并把成功轨迹存入向量化经验池；推理时，它根据任务相似度检索最相关的历史成功案例作为示范，同时将提炼好的规则注入提示词，从而在单次尝试中做出更明智的决策——整个过程完全不需要更新模型参数。在三个差异显著的基准域上，ExpeL持续超越ReAct、Reflexion等强基线，并展现出从源任务向目标任务正向迁移的能力。这项工作的意义在于，它为只能通过API访问的闭源模型（如GPT-4）提供了一条无需触碰模型权重、却能让智能体随经验积累持续变强的实用路径。
原文 arXiv:2308.10144;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.10144v3