API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Minghao Li1∗, Yingxiu Zhao2, Bowen Yu1, Feifan Song3 Hangyu Li1, Haiyang Yu1, Zhoujun Li4, Fei Huang1, Yongbin Li1† 1Alibaba Group, 2Hong Kong University of Science and Technology, 3Peking University, 4Shenzhen Intelligent Strong Technology Co., Ltd, {lmh397008, yubowen.ybw, hangyu.lhy, yifei.yhy, f.huang, Equal Contributions Corresponding author.
Abstract
Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs’ ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs’ capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca’s tool utilization performan
中文速览
让大型语言模型（LLM）学会调用外部工具（API）是让它突破自身知识局限的关键一步，但学界此前既缺乏系统评测手段，也没有大规模训练数据。研究团队为此构建了 API-Bank 这一专项基准，包含 73 个可实际运行的 API、314 段人工标注的对话和 753 次 API 调用，用于衡量模型在"直接调用""检索后调用""规划-检索-调用"三个难度层级上的表现；同时提出了一种多智能体（Multi-agent）自动数据生成方法，以仅为人工标注 2% 的成本生产了涵盖 2138 个 API、1888 段对话的训练集，并据此微调出工具增强模型 Lynx。实验表明，GPT-4 在规划能力上最强，但仍有较大提升空间；基于 Alpaca-7B 训练的 Lynx 在三项 API 使用能力上平均提升超过 26 个百分点，接近 GPT-3.5 水平。这项工作首次提供了一套可执行、可复现的工具调用评测体系，为未来研究工具增强型语言模型奠定了数据与方法基础。
原文 arXiv:2304.08244;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.08244v2