LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Shilong Liu♠♠\spadesuit∗, Hao Cheng♣♣\clubsuit, Haotian Liu♢♢\diamondsuit∗, Hao Zhang♡♡\heartsuit∗, Feng Li♡♡\heartsuit∗, Tianhe Ren▶, Xueyan Zou♢♢\diamondsuit∗, Jianwei Yang♣♣\clubsuit, Hang Su♠♠\spadesuit, Jun Zhu♠♠\spadesuit, Lei Zhang▶, Jianfeng Gao♣♣\clubsuit, Chunyuan Li♣♣\clubsuit \faFlag ♠♠\spadesuitDept. of Comp. Sci.、Tech., Institute for AI, BNRist, Tsinghua University ♣♣\clubsuitMicrosoft Research, Redmond ♢♢\diamondsuitUniversity of Wisconsin-Madison ♡♡\heartsuitHKUST ▶IDEA Research ∗Work performed during an internship at Microsoft \faFlag Project Lead https://llava-vl.github.io/llava-plus/
Abstract
This paper presents LLaVA-Plus (Large Language and Vision Assistants that Plug and Learn to Use Skills), a general-purpose multimodal assistant trained using an end-to-end approach that systematically expands the capabilities of large multimodal models (LMMs). LLaVA-Plus maintains a skill repository that contains a wide range of vision and vision-language pre-trained models (tools), and is able to activate relevant tools, given users’ multimodal inputs, to compose their execution results on the fly to fulfill many real-world tasks. To acquire the ability of using tools, LLaVA-Plus is trained on multimodal instruction-following data that we have curated. The training data covers many tool use examples of visual understanding, generation, external knowledge retrieval and their compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in existing capabilities, and exhibits many new capabilities. Compared with tool-augmented LLMs, LLaVA-Plus is distinct in that the image query is directly grounded in and actively engaged throughout the entire human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios.
中文速览
大规模多模态模型（LMM）在视觉理解上已取得不错成绩，但面对图像生成、精细分割、外部知识检索等复杂现实任务时往往力不从心，而纯靠提示词驱动工具调用的方案又不够稳健。LLaVA-Plus 的思路是把"端到端训练"和"工具链调用"两者的优点合二为一：它维护一个涵盖检测、分割、生成、知识检索等多类专项模型的技能库（skill repository），并通过专门构造的多模态指令跟随数据对模型进行微调，让模型学会在完整的人机对话过程中主动选择并组合调用合适的工具，而非仅在触发工具时才考虑视觉信息。实验结果表明，LLaVA-Plus 在已有基准上超越了原版 LLaVA，还在面向真实场景的 VisiT-Bench 上达到新的最优水平，并涌现出原先无法完成的新任务能力。这项工作意义在于提供了一条可持续扩展的路径——只需收集新工具的指令数据并补充微调，就能低成本地为多模态助手"插件式"升级新技能，推动通用多模态 AI 助手的落地。
原文 arXiv:2311.05437;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.05437v1