Aha.
正在载入中英对照阅读…

arXiv:2611.18533 · 中英对照阅读

ModHarness-RL:将智能体运行框架重新构想为可学习、可演进的智能体强化学习程序

ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL

Zelin He、Siyu Zhu、Jacob Helwig、Yujian Liu、Li An、Rohit Jain、Shiyu Chang、Alborz Geramifard Website

中文速览

智能体强化学习常用一套固定的“脚手架”来整理对话、提示模型,但不同任务和训练阶段需要的脚手架并不一样。研究者提出 ModHarness-RL,把交互记录整理成带类型的信息块,再让程序自动压缩历史、识别反复出现的错误并提供针对性提示;这些程序根据训练轨迹生成和筛选,提示规则还会随模型进步而更新。实验显示,在代码定位、家居任务和数据库问答中,这种方法大多能提升未见任务上的表现,代码定位尤其比固定脚手架高出 14% 到 24%。这说明智能体不只模型本身可以通过强化学习变强,围绕模型的工作方式也能从数据中学出来,并随训练一起改进。

摘要

智能体强化学习(RL)日益依赖智能体运行框架,即模型周围的支架,它决定模型能看到自身交互的哪些内容,以及会接收何种指导。现有的大多数方法都在人工编写且训练过程中始终固定不变的智能体运行框架下进行训练。然而,有效的智能体运行框架取决于领域、数据和模型,并会随着模型训练不断变化。我们提出 ModHarness-RL,将智能体运行框架重新构想为从数据中学习、并随模型通过强化学习训练而演进的程序。ModHarness-RL 将交互记录为结构化状态,即一组类型化数据块;智能体运行框架的两个核心组件则是作用于这些数据块的程序:压缩程序(Compression program)将累积的历史记录整合到模型的上下文中,引导程序(Steering program)则在行为偏离时提供指导。在强化学习训练期间,系统根据近期轨迹提出候选程序,经执行验证后,再在过去的轨迹上重放,使智能体运行框架能够跟踪模型行为的变化。我们深入研究了 CodeSearch,并进一步分析了 ALFWorld 和 BIRD SQL,所用模型规模从 1.5B 到 8B。在大多数设置下,所提出的压缩和引导机制都能提升留出集性能,优于固定运行框架基线。这些结果表明,智能体强化学习的运行框架可以从任务数据中学习,并随模型共同演进。

Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model’s context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model’s behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task’s data and evolved with the model.

术语表

Agentic reinforcement learning (agentic RL)
智能体强化学习
harness
智能体运行框架
ModHarness-RL
ModHarness-RL
structured State
结构化状态
typed blocks
类型化数据块
Compression program
压缩程序
Steering program
引导程序
Replay mechanism
重放机制
multi-turn agent
多轮智能体
large language model (LLM)
大语言模型
policy
策略
trajectory
轨迹
context budget
上下文预算
fixed-harness baseline
固定运行框架基线
held-out performance
留出集性能
CodeSearch
CodeSearch
ALFWorld
ALFWorld
BIRD SQL
BIRD SQL
GRPO
GRPO
memory
记忆
summarization
摘要
pruning
剪枝
guardrails
安全约束
online data
在线数据
candidate program
候选程序
tool call
工具调用
observation
观测
reward
奖励