StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Zhicheng Guo1,2121,21 , 2, Sijie Cheng1,2,31231,2,31 , 2 , 3, Hao Wang4444, Shihao Liang5555, Yujia Qin1111, Peng Li2222, Zhiyuan Liu1111, Maosong Sun1111, Yang Liu1,2,6 1111Dept. of Comp. Sci.、Tech., Institute for AI, Tsinghua University, Beijing, China 2222Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China 333301.AI 4444Google 5555The University of Hong Kong 6666Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China {guo-zc21,
Abstract
Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.
中文速览
大规模工具调用基准测试长期面临两大难题：真实在线API会随时失效或需要重新授权，而自动评估器（GPT-3.5）又无法准确判断任务是否可解，导致不同时间、不同批次的实验结果根本无法对比。为此，研究团队在ToolBench基础上构建了StableToolBench，核心是一套虚拟API服务器——先用缓存系统存储历史真实调用结果，缓存未命中时再调用GPT-4-Turbo根据API文档和少量真实示例模拟返回值，确保模型始终能得到稳定响应。同时，他们将评估器升级为GPT-4，并设计了"可解通过率（SoPR）"和"可解胜率（SoWR）"两个新指标，剔除不可解任务的干扰和随机判断带来的噪声。实验表明，新基准在多次重复评测中表现高度一致，且模拟API在真实性、多样性和文档遵循度上均接近真实API，为工具学习研究提供了一个兼顾规模与稳定性的可靠评测平台。
原文 arXiv:2403.07714;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2403.07714v5