Aha.
正在载入中英对照阅读…

arXiv:2605.08693 · 中英对照阅读

SkillMaster:迈向大语言模型智能体的自主技能掌握

SkillMaster: Toward Autonomous Skill Mastery in LLM Agents

Min Yang、Jinghua Piao、Xu Xia、Xiaochong Lan、Jiaju Chen、Yongshun Gong、Yong Li

中文速览

让智能体不只是调用现成技能,而是能从做任务的经历中自己总结、修改并筛选技能,是提升复杂任务表现的关键难题。SkillMaster让智能体每次完成任务后回看过程,决定新增、改进还是保留技能;如果提出了修改,还会在相似任务上对比修改前后的效果,并用分别处理任务执行与技能管理奖励的训练方法来稳定学习。在ALFWorld、WebShop和SearchQA测试中,它整体优于对比方法,也能发现技能哪里失灵、据此改进,并把改进迁移到后续任务。这个结果说明,技能不必只是外部提供的工具,也可以成为智能体从经验中不断成长的能力。

摘要

技能为提升大语言模型智能体在复杂任务上的表现提供了有效机制,但在现有智能体框架中,技能的创建、完善与选择通常由外部教师、人工设计的规则或辅助模块主导。因此,技能仍然是供智能体调用的外部资源,而非智能体能够通过经验开发、调整并内化的能力。为赋予大语言模型智能体自主掌握技能的能力,我们提出了 SkillMaster,这是一种训练框架,旨在教会智能体创建新技能、完善现有技能,并在任务求解过程中选择已积累的技能。这一能力通过三项关键设计实现。首先,我们通过轨迹信息引导的技能审查训练智能体,使其学习根据已完成回合中的证据提出、更新或保留技能。其次,我们设计了评估每项候选技能编辑在相关探测任务上的反事实效用的方法,为训练技能编辑决策提供直接的学习信号。第三,我们提出 DualAdv-GRPO,分别估计任务求解动作和技能编辑决策的优势,从而稳定任务求解与技能管理的联合训练。在 ALFWorld、WebShop 和 SearchQA 上开展的实验表明,在所有对比方法中,SkillMaster 的整体表现最佳。进一步分析揭示了智能体能力的显著转变:通过 SkillMaster 训练的智能体能够识别技能失效,根据轨迹证据完善程序性知识,并且仅需对技能库进行有限编辑,就能将改进迁移到未来任务中。总之,SkillMaster 推动大语言模型智能体超越单纯的技能使用,使其成为能够开发、调整并运用自身技能体系的自我改进型智能体。00footnotetext:我们的代码已发布于 https://github.com/sduyangmin/Skill-Master。

Skills provide an effective mechanism for improving LLM agents on complex tasks, yet in existing agent frameworks, their creation, refinement, and selection are typically governed by external teachers, hand-designed rules, or auxiliary modules. As a result, skills remain external resources to be invoked, rather than capabilities that agents can develop, adapt, and internalize through experience. To endow LLM agents with autonomous skill mastery, we propose SkillMaster, a training framework that teaches agents to create new skills, refine existing skills, and select accumulated skills during task solving. This capability is achieved through three key designs. First, we train agents through trajectory-informed skill review, teaching agents to propose, update, or retain skills based on evidence from completed episodes. Second, each candidate skill edit is designed to be evaluated by its counterfactual utility on related probe tasks, providing a direct learning signal for training skill-editing decisions. Third, we introduce DualAdv-GRPO, which separately estimates advantages for task-solving actions and skill-editing decisions, stabilizing joint training across task solving and skill management. Experiments on ALFWorld, WebShop, and SearchQA show that SkillMaster achieves the best overall performance among all compared methods. Further analysis reveals a marked shift in agent capability: agents trained with SkillMaster can identify skill failures, refine procedural knowledge from trajectory evidence, and transfer improvements to future tasks with limited skill-bank edits. Overall, SkillMaster moves LLM agents beyond mere skill use toward self-improving agents capable of developing, adapting, and applying their own skill repertoires. 00footnotetext: Our code is released at https://github.com/sduyangmin/Skill-Master.

术语表

LLM agents
大语言模型智能体
skills
技能
skill mastery
技能掌握
SkillMaster
SkillMaster
skill bank
技能库
trajectory-informed skill review
轨迹信息引导的技能审查
trajectory
轨迹
episode
回合
skill management
技能管理
skill-editing decisions
技能编辑决策
counterfactual utility
反事实效用
probe tasks
探测任务
downstream utility reward
下游效用奖励
DualAdv-GRPO
DualAdv-GRPO
GRPO
GRPO
PPO
PPO
advantage normalization
优势归一化
reinforcement learning
强化学习
policy
策略
skill retrieval
技能检索
tool-integrated reasoning
工具集成推理
ALFWorld
ALFWorld
WebShop
WebShop
SearchQA
SearchQA
OpenClaw
OpenClaw
Claude Code
Claude Code