MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
Chenyu Wang1, Weixin Luo2, Sixun Dong1, Xiaohua Xuan3, Zhengxin Li1, Lin Ma2, Shenghua Gao4 1ShanghaiTech University, 2Meituan, 3UniDT Technology, 4University of Hong Kong
Abstract
Recently, the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However, the current LLMs’ ability to perceive tool use is limited to a single text query, which may result in ambiguity in understanding the users’ real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore, in this paper, we propose MLLM-Tool, a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model’s capability, we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions, which provides more potential solutions for t
中文速览
大模型（LLM）做工具调用时，仅靠文字描述往往说不清用户真正想要什么——比如"根据图片生成一张图"这句话，不看图就无法判断该用姿态引导还是轮廓引导的生成工具。为此，研究者提出了 MLLM-Tool 系统，将多模态编码器（处理图像、视频、音频）与开源大语言模型结合，让模型直接"看到"或"听到"用户指令中的非文字信息，从而准确选择匹配的外部 API 工具。配套地，他们从 HuggingFace 平台筛选出 932 个高质量模型，构建了名为 ToolMMBench 的多模态工具选择基准数据集，其一大特色是同一条指令可对应多个合理答案（相同或近似功能的工具），更贴近真实使用场景。实验表明，在 ToolMMBench 上微调后，MLLM-Tool 的工具选择准确率达到 88.19%，证明引入多模态感知能力可以有效消除纯文本指令的歧义，为构建更智能的工具调用代理提供了新思路。
原文 arXiv:2401.10727;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2401.10727v3