Introspective Tips: Large Language Model for In-Context Decision Making
Liting Chen1, Lu Wang1, Hang Dong1, Yali Du2, Jie Yan1, Fangkai Yang1, Shuang Li3, Pu Zhao1, Si Qin1, Saravan Rajmohan1, Qingwei Lin1, Dongmei Zhang1
Abstract
The emergence of large language models (LLMs) has substantially influenced natural language processing, demonstrating exceptional results across various tasks. In this study, we employ “Introspective Tips” to facilitate LLMs in self-optimizing their decision-making. By introspectively examining trajectories, LLM refines its policy by generating succinct and valuable tips. Our method enhances the agent’s performance in both few-shot and zero-shot learning situations by considering three essential scenarios: learning from the agent’s past experiences, integrating expert demonstrations, and generalizing across diverse games. Importantly, we accomplish these improvements without fine-tuning the LLM parameters; rather, we adjust the prompt to generalize insights from the three aforementioned situations. Our framework not only supports but also emphasizes the advantage of employing LLM in in-contxt decision-making. Experiments involving over 100 games in TextWorld illustrate the superior performance of our approach.
中文速览
让大语言模型（LLM）在文字游戏等序列决策任务中自主优化策略，是当前的核心挑战——模型容易产生幻觉、缺乏领域知识，而已有的"自我反思"方法又只能针对单个智能体的失败经验，泛化能力有限。研究者提出了"内省提示"（Introspective Tips）方法：让LLM回顾自身、专家或其他游戏环境的历史轨迹，从中提炼出简洁、高层次的行动建议，再把这些"小贴士"动态注入到提示词中来指导后续决策，全程无需对模型参数进行微调。在超过100款TextWorld文字游戏上的实验表明，仅凭48条轨迹生成的提示，LLM智能体就能在最高难度的未见游戏中超越训练了10万轮的深度强化学习基线。这项工作证明，通过精心设计的提示工程而非昂贵的模型训练，LLM就能具备跨场景泛化的自我进化决策能力，为低样本消耗、高可解释性的智能体开发提供了新思路。
原文 arXiv:2305.11598;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.11598v1