Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models
Cheng-Yu Hsieh , Si-An Chen , Chun-Liang Li3, Yasuhisa Fujii4, Alexander Ratner1, Chen-Yu Lee3, Ranjay Krishna1∗, Tomas Pfister3∗ 1University of Washington, 2National Taiwan University, 3Google Cloud AI Research, 4Google Research
Abstract
Today, large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool’s usage. Unfortunately, demonstrations are hard to acquire, and can result in undesirable biased usage if the wrong demonstration is chosen. Even in the rare scenario that demonstrations are readily available, there is no principled selection protocol to determine how many and which ones to provide. As tasks grow more complex, the selection search grows combinatorially and invariably becomes intractable. Our work provides an alternative to demonstrations: tool documentation. We advocate the use of tool documentation—descriptions for the individual tool usage—over demonstrations. We substantiate our claim through three main empirical findings on $6$ tasks across both vision and language modalities. First, on existing benchmarks, zero-shot prompts with only tool documentation are sufficient for eliciting proper tool usage, achieving performance on par with few-shot prompts. Second, on a newly collected realistic tool-use dataset with hundreds of available tool APIs, we show that tool documentation is significantly more valuable than demonstrations, with zero-shot documentatio
中文速览
大型语言模型（LLM）要学会调用外部工具时，通常依赖人工准备的少量示例（few-shot demonstrations），但示例难以获取、容易引入偏差，工具数量一多就面临组合爆炸。这项研究提出用"工具文档"（tool documentation）替代示例——就像工匠靠阅读说明书而非亲眼观摩就能上手新工具，只需把每个工具的功能描述和调用方式提供给 LLM，就足以让它规划出正确的工具调用方案。在涵盖视觉和语言的 6 项任务上，实验表明：仅凭工具文档的零样本提示，性能与少样本示例相当甚至更优；面对含 200 个 API 的大规模真实工具集时，文档的优势更加显著，远超没有文档的少样本方案。更令人惊喜的是，借助 GroundingDINO、Stable Diffusion、XMem 和 SAM 的文档，LLM 能够自动"重新发明"出 Grounded-SAM 和 Track Anything 这两个刚发布的前沿系统的功能，说明零样本工具使用有望驱动自动知识发现，为构建可扩展、免标注的 LLM 工具生态提供了切实可行的路径。
原文 arXiv:2308.00675;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.00675v1