ControlLLM: Augment Language Models with Tools by Searching on Graphs
Zhaoyang Liu*1,2absent12{}^{*1,2}start_FLOATSUPERSCRIPT * 1 , 2 end_FLOATSUPERSCRIPT Zeqiang Lai*2absent2{}^{*2}start_FLOATSUPERSCRIPT * 2 end_FLOATSUPERSCRIPT Zhangwei Gao22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Erfei Cui22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Ziheng Li33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT Xizhou Zhu2,323{}^{2,3}start_FLOATSUPERSCRIPT 2 , 3 end_FLOATSUPERSCRIPT Lewei Lu44{}^{4}start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPT Qifeng Chen11{1}~{}1✉ Yu Qiao22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Jifeng Dai2,323{}^{2,3}start_FLOATSUPERSCRIPT 2 , 3 end_FLOATSUPERSCRIPT Wenhai Wang22{2}~{}2✉ 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTThe Hong Kong University of Science and Technology 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTOpenGVLab, Shanghai AI Laboratory 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPTTsinghua University 44{}^{4}start_FLOATSUPERSCRIPT 4 end_FLOATSUPERSCRIPTSenseTime https://github.com/OpenGVLab/ControlLLM
Abstract
We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks. Despite the remarkable performance of LLMs, they still struggle with tool invocation due to ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling. To overcome these challenges, our framework comprises three key components: (1) a task decomposer that breaks down a complex task into clear subtasks with well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG) paradigm that searches the optimal solution path on a pre-built tool graph, which specifies the parameter and dependency relations among different tools; and (3) an execution engine with a rich toolbox that interprets the solution path and runs the tools efficiently on different computational devices. We evaluate our framework on diverse tasks involving image, audio, and video processing, demonstrating its superior accuracy, efficiency, and versatility compared to existing methods.
中文速览
现有的工具增强型大语言模型在面对复杂多模态任务时，常因提示模糊、工具选择错误和参数分配不准确而频繁失败。ControlLLM 提出了一套三阶段框架来解决这一问题：首先由任务分解器把用户请求拆成有明确输入输出的子任务，再通过"图上思维"（Thoughts-on-Graph，ToG）范式在预先构建好的工具依赖图上用搜索算法找到最优工具调用路径，最后由执行引擎并行调度各工具完成任务。在涵盖图像、音频、视频处理的基准测试上，ControlLLM 在困难任务的整体方案评估中成功率达到 93%，而最强基线方法仅为 59%。这项工作的意义在于突破了链式或树式规划的拓扑局限，让大语言模型能够可靠地驾驭结构复杂的多模态工具调用，同时无需重新训练模型即可灵活扩展工具库。
原文 arXiv:2310.17796;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.17796v3