MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
Yue Huang1111Lichao Sun and Yue Huang are co-corresponding authors: 222Visiting Students at LAIR Lab, Lehigh University., Jiawen Shi2, Yuan Li3, Chenrui Fan2, Siyuan Wu2, Qihui Zhang1222Visiting Students at LAIR Lab, Lehigh University. Yixin Liu1, Pan Zhou2, Yao Wan2, Neil Zhenqiang Gong4, Lichao Sun1111Lichao Sun and Yue Huang are co-corresponding authors: Lehigh University1 Huazhong University of Science and Technology2 University of Cambridge3 Duke University4
Abstract
Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with simil
中文速览
现有的大模型评测基准主要关注"如何用工具"，却忽视了更基础的两个问题：模型知不知道自己需要借助工具，以及面对一堆候选工具时能不能选对。为此，作者构建了MetaTool评测框架，核心是一个包含21,127条用户查询的ToolE数据集，覆盖单工具和多工具场景，并设计了工具使用意识、以及四类工具选择子任务（相似选项辨别、特定场景适配、可靠性判断、多工具推理）。对八款主流大模型的实验表明，绝大多数模型仍难以准确判断何时该用工具、用哪个工具，与真正意义上的智能体还有明显差距。这项工作填补了工具调用能力评测中"决策前端"的空白，同时为工具开发者提供了如何匹配下游模型来优化工具描述的实用建议。
原文 arXiv:2310.03128;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.03128v6