Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark
Mengsong Wu Tong Zhu Han Han Chuanyuan Tan Xiang Zhang Wenliang Chen Soochow University, Shizi Street 1, 215006 Suzhou, China
Abstract
This paper presents a new tool learning dataset Seal-Tools, which contains self-instruct API-like tools. Seal-Tools not only offers a large number of tools, but also includes instances which demonstrate the practical application of tools. Seeking to generate data on a large scale while ensuring reliability, we propose a self-instruct method to generate tools and instances, allowing precise control over the process. Moreover, our Seal-Tools contains hard instances that call multiple tools to complete the job, among which some are nested tool callings. For precise and comprehensive evaluation, we use strict format control and design three metrics from different dimensions. Therefore, Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on Seal-Tools. The results show that current systems are far from perfect. The code, data and experiment results are available at https://github.com/fairyshine/Seal-Tools .
中文速览
大型语言模型（LLM）作为智能代理使用工具时，现有数据集要么规模太小、要么评估不够精确，难以有效训练和衡量模型的工具调用能力。为此，研究团队提出了一套"自我指令"（self-instruct）数据构建方法，先让 LLM 生成分层领域字段，再在字段约束下生成 4,076 个工具和逾万条调用实例，并通过多步生成与格式校验大幅减少幻觉错误，同时还构造了包含嵌套工具调用的高难度样本。最终发布的 Seal-Tools 数据集在格式输出、工具选择、参数填写三个维度上设计了严格的自动化评估指标，对主流 LLM 和微调模型的测试结果显示，当前系统离完美还差得很远，尤其在嵌套调用场景下表现明显不足。这项工作既为代理微调提供了高质量、大规模的数据资源，也为工具调用能力的精细化评估提供了可复现的基准。
原文 arXiv:2405.08355;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2405.08355v1