Skill-R1: Agent Skill Evolution via Reinforcement Learning

摘要

Agentic large language models often rely on skills, reusable natural-language procedures that guide planning, action, and tool use. In practice, however, skills are typically improved through prompt engineering or by aligning the task LLM itself to revised skills, which is costly, model-specific, and often infeasible for closed-source models. Meanwhile, skill optimization is not a one-step prompt engineering problem, but a recurrent process with two coupled levels of credit assignment. A useful skill must improve rollout quality under the current conditioning, while a useful revision must turn observed successes and failures into a better skill for the next round. We therefore formulate skill evolution as a bi-level optimization problem over rollout selection within each generation and skill improvement across generations. We propose Skill-R1, a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards. Rather than updating the task LLM, Skill-R1 trains a lightweight skill generator that conditions on the task context, prior rollouts, and their verified outcomes to produce skills that steer a frozen task LLM. This design preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates. Skill-R1 proceeds over multiple generations. At each generation, the current skill induces rollouts from the task LLM, whose verified outcomes are fed back to produce the next revision. To optimize the skill generator over this recurrent process, we introduce a bi-level group-relative policy optimization objective with intra-generation and inter-generation advantages. The intra-generation term compares rollouts under shared skill conditioning, while the inter-generation term rewards revisions that improve the induced behavior over successive generations. Together, these terms provide a principled objective for directional skill evolution rather than one-shot or heuristic self-refinement. Empirical evaluations suggest that Skill-R1 achieves consistent improvements over both no-skill baselines and standard GRPO across benchmarks with verifiable rewards. The gains are particularly strong on complex, multi-step tasks, where single-pass refinement methods struggle.