GTA: A Benchmark for General Tool Agents
Jize Wang1,2 Zerun Ma2 Yining Li2 Songyang Zhang2 Cailian Chen1 Kai Chen2∗ Xinyi Le1 1Shanghai Jiao Tong University 2Shanghai AI Laboratory Corresponding Authors.
Abstract
Significant focus has been placed on integrating large language models (LLMs) with various tools in developing general-purpose agents. This poses a challenge to LLMs’ tool-use capabilities. However, there are evident gaps between existing tool-use evaluations and real-world scenarios. Current evaluations often use AI-generated queries, single-step tasks, dummy tools, and text-only interactions, failing to effectively reveal the agents’ real-world problem-solving abilities. To address this, we propose GTA, a benchmark for General Tool Agents, featuring three main aspects: (i) Real user queries: human-written queries with simple real-world objectives but implicit tool-use, requiring the LLM to reason the suitable tools and plan the solution steps. (ii) Real deployed tools: an evaluation platform equipped with tools across perception, operation, logic, and creativity categories to evaluate the agents’ actual task execution performance. (iii) Real multimodal inputs: authentic image files, such as spatial scenes, web page screenshots, tables, code snippets, and printed/handwritten materials, used as the query contexts to align with real-world scenarios closely. We design 229 real-world
中文速览
大量研究聚焦于让大语言模型（LLM）调用外部工具以构建通用智能体，但现有评测与真实使用场景之间存在明显鸿沟——测试问题由AI生成、工具是虚拟模拟的、交互仅限文字，根本无法反映模型在现实中解决问题的真实水平。为此，研究者提出了GTA（General Tool Agents）基准，核心特点是"三真"：由人工撰写、目标明确但工具调用步骤隐式的真实用户查询；横跨感知、操作、逻辑、创作四类共14款可实际运行的真实工具；以及场景照片、网页截图、表格、代码片段、手写材料等真实多模态图像输入。研究团队设计了229个任务及对应可执行工具调用链，对16款主流LLM进行了系统评测，结果显示GPT-4完成率不足50%，大多数模型低于25%。这一发现精准定位了当前LLM在真实场景下工具调用能力的瓶颈，为通用工具智能体的下一步研发方向提供了清晰参照。
原文 arXiv:2407.08713;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2407.08713v2