GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Rui Yang1111Equal contribution. ‡‡\ddagger Work done during an internship at Tencent. , Lin Song 222Corresponding author., Yanwei Li3, Sijie Zhao2, Yixiao Ge2, Xiu Li1, Ying Shan2 1Tsinghua Shenzhen International Graduate School, Tsinghua University 2Tencent AI Lab 3Chinese University of Hong Kong
Abstract
This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools. Advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools. It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation. Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools. The code and demo are available at https://github.
中文速览
让大型语言模型（LLM）学会调用图像识别、图像生成等多模态工具，过去只有 ChatGPT、GPT-4 这类闭源商业模型才能做到，而这些模型既昂贵又不对外开放。GPT4Tools 用"自指令（self-instruct）"的思路解决这一问题：先让 ChatGPT 结合真实图像内容自动生成大量与工具调用相关的指令数据集（约 41K 条），再通过轻量级的低秩适配（LoRA）微调 Vicuna、OPT 等开源模型，使它们学会何时、如何调用视觉工具来完成目标检测、图像生成、视觉问答等任务。实验结果显示，微调后的 Vicuna-13B 在工具调用成功率上比直接用 GPT-3.5 提示工程高出 9.3 个百分点，甚至对从未见过的新工具也具备可观的零样本泛化能力。这项工作的意义在于，它为普通研究者提供了一条低成本路径，无需依赖封闭的商业 API，就能让开源语言模型具备强大的多模态工具使用能力。
原文 arXiv:2305.18752;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.18752v1