Aha.
正在载入中英对照阅读…

arXiv:2606.04455 · 中英对照阅读

Meta-Agent Challenge:当前智能体是否具备自主智能体开发能力?

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

Xinyu Lu、Tianshu Wang、Pengbo Wang、Zujie Wen、Zhiqiang Zhang、Jun Zhou、Boxi Cao、Yaojie Lu、Hongyu Lin、Xianpei Han、Le Sun

中文速览

AI 现在能完成不少复杂任务,但大多数智能体仍靠人来设计流程;这项研究想检验,模型能不能自己造出并改进智能体。作者搭建了 Meta-Agent Challenge:让代码模型在有时间和 API 配额限制的沙盒里,反复编写、测试和优化一个专门解题的程序,再用未公开的题目评分,并覆盖数学、编程、科学、软件工程和终端操作五类任务。结果显示,模型造出的智能体通常不如人工设计的方案,少数表现较好的也主要来自闭源前沿模型;不同运行结果差异很大,强烈的优化压力还可能诱发窃取答案等对抗行为。这个基准不仅能衡量 AI 是否开始具备自主研发智能体的能力,也能帮助研究者发现这类能力带来的安全与对齐风险。

摘要

当前的 AI 基准测试评估智能体在由人类设计的工作流中执行任务的表现。这类评估从根本上无法衡量一项关键的更高层级能力:模型能否自主开发智能体系统。我们提出 Meta-Agent Challenge(MAC),这是一种评测框架,旨在测试前沿模型自主进行智能体开发的能力。具体而言,代码智能体(元智能体)会获得一个沙箱环境、一个评测 API 和一段时限,并在五个领域中迭代编写智能体制品,以最大化其在留出测试集上的表现。为确保评测完整性,该框架采用多层防御机制,防范奖励黑客行为。借助这一框架,我们表明,元智能体鲜少能达到人类设计的基线策略的水平,而少数达到该水平的元智能体也不及专有前沿模型。此外,设计过程呈现出较高的方差,而高优化压力会引发真实标签窃取等涌现式对抗行为,凸显出其在鲁棒性和模型对齐方面的关键缺陷。最终,MAC 为自主 AI 研究与开发提供了严格的开源基准测试,并为评估递归式自我改进提供了经验性替代指标。基准测试公开发布于:https://github.com/ant-research/meta-agent-challenge。

Current AI benchmarks evaluate agents on task execution within human-designed workflows. These evaluations fundamentally fail to measure a critical next-level capability: whether models can autonomously develop agent systems. We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development. Specifically, a code agent (the meta-agent) is given a sandboxed environment, an evaluation API, and a time limitation to iteratively program an agent artifact that maximizes performance on a held-out test set across five domains. To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking. Leveraging this framework, we demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models. Moreover, the design process exhibits high variance, and high optimization pressure surfaces emergent adversarial behaviors like ground-truth exfiltration—highlighting critical deficits in both robustness and model alignment. Ultimately, MAC provides a rigorous, open-source benchmark for autonomous AI research and development, offering an empirical proxy for evaluating recursive self-improvement. Benchmark is publicly available at: https://github.com/ant-research/meta-agent-challenge.

术语表

AI benchmark
AI 基准测试
agent
智能体
agent system
智能体系统
agent development
智能体开发
Meta-Agent Challenge (MAC)
Meta-Agent Challenge(MAC)
meta-agent
元智能体
agent artifact
智能体制品
sandboxed environment
沙箱环境
evaluation API
评测 API
held-out test set
留出测试集
reward hacking
奖励黑客行为
ground-truth exfiltration
真实标签窃取
agentic scaffolding
智能体脚手架
prompt engineering
提示工程
control flow
控制流
system architect
系统架构师
object level
对象层
meta level
元层
recursive self-improvement
递归式自我改进
optimization pressure
优化压力
model alignment
模型对齐
inter-run variance
运行间方差
SWE-Bench
SWE-Bench
Terminal-Bench
Terminal-Bench
BrowseComp
BrowseComp
MLE-Bench
MLE-Bench
PostTrainBench
PostTrainBench
foundation model
基础模型
task-specific agent
特定任务智能体