Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
Chuofan Ma1 Yi Jiang2† Jiannan Wu1 Zehuan Yuan2 Xiaojuan Qi1 Work done during Chuofan’s internship at ByteDance.Corresponding authors
Abstract
We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning and visual grounding. Such capabilities are built upon a localized visual tokenization mechanism, where an image input is decomposed into regions of interest and subsequently encoded into region tokens. By integrating region tokens into user instructions and model responses, we seamlessly enable Groma to understand user-specified region inputs and ground its textual output to images. Besides, to enhance the grounded chat ability of Groma, we curate a visually grounded instruction dataset by leveraging the powerful GPT-4V and visual prompting techniques. Compared with MLLMs that rely on the language model or external module for localization, Groma consistently demonstrates superior performances in standard referring and grounding benchmarks, highlighting the advantages of embedding localization into image tokenization. Project page: https://groma-mllm.github.io/.
中文速览
当前的多模态大语言模型（MLLM）普遍缺乏精准的视觉定位能力，要么让语言模型直接输出坐标（计算代价高且不擅长），要么借助SAM等外部模块（需要二次处理图像、增加延迟）。Groma提出了一种全新思路：在图像分词（tokenization）阶段就完成定位，用一个区域提议器（region proposer）自动发现感兴趣区域，再用区域编码器将这些区域编码成"区域token"，直接嵌入用户指令和模型回复中，从而把"指代输入"和"定位输出"统一到同一套机制里。在数据层面，作者还利用GPT-4V结合视觉提示技术构建了首个同时包含视觉和文本提示的有根据对话数据集Groma Instruct用于指令微调。实验表明，Groma在标准视觉指代和视觉定位基准上全面超越同类方法，在具有挑战性的LVIS目标定位评测上领先超过10% AR，同时保持了良好的图像级理解与推理能力——这说明将定位能力内嵌到视觉分词过程是一条兼顾精度与效率的有效路径。
原文 arXiv:2404.13013;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2404.13013v1