On the Tool Manipulation Capability of Open-source Large Language Models
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian Zhang SambaNova Systems, Inc. Palo Alto, CA, USA
Abstract
Recent studies on software tool manipulation with large language models (LLMs) mostly rely on closed model APIs. The industrial adoption of these models is substantially constrained due to the security and robustness risks in exposing information to closed LLM API services. In this paper, we ask can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation, with practical amount of human supervision. By analyzing common tool manipulation failures, we first demonstrate that open-source LLMs may require training with usage examples, in-context demonstration and generation style regulation to resolve failures. These insights motivate us to revisit classical methods in LLM literature, and demonstrate that we can adapt them as model alignment with programmatic data generation, system prompts and in-context demonstration retrievers to enhance open-source LLMs for tool manipulation. To evaluate these techniques, we create ToolBench111Available at https://github.com/sambanova/toolbench, a tool manipulation benchmark consisting of diverse software tools for real-world tasks. We demonstrate that our techniques can boost leading open-source LLMs by up to $9
中文速览
开源大模型在工具调用（tool manipulation）任务上远落后于GPT-4这类闭源模型——差距高达70%以上，根源在于三个具体短板：不会准确识别API名称、无法正确填充API参数、以及频繁生成无法执行的自然语言输出。针对这三个问题，研究者提出了一套以极少人工标注为代价的增强方案：用程序化批量生成的模板数据对开源模型做指令微调（model alignment）、在推理时用语义检索器动态引入示范样例（in-context demonstration retriever）、并设计系统提示词（system prompt）约束输出格式。为系统评测该方案，作者还构建了ToolBench基准，涵盖从操控Google表格到控制机器人的8项真实任务。实验表明，这套方法最高可将开源模型的成功率提升90%，在8项任务中的4项达到与GPT-4持平甚至更优的水平，且每个工具平均只需一名开发者花费约一天时间完成数据标注，具备切实的工业落地价值。
原文 arXiv:2305.16504;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.16504v1