Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
Shilong Liu1,2 This work was done when Shilong Liu, Feng Li, Hao Zhang, Jie Yang, and Qing Jiang were interns at IDEA. Zhaoyang Zeng2 Tianhe Ren2 Feng Li2, 3 Hao Zhang2, 3 Jie Yang2, 4 Qing Jiang2, 6 Chunyuan Li5 Jianwei Yang5 Hang Su1 Jun Zhu1⋆⋆ Lei Zhang2 Corresponding authors.
Abstract
In this paper, we develop an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for modalities fusion. We first pre-train Grounding DINO on large-scale datasets, including object detection data, grounding data, and caption data, and evaluate the model on both open-set object detection and referring object detection benchmarks. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO zero-shot111In this paper, ‘zero-shot’ refers to scenarios where the training split of the test dataset is not utilized in the training process.
中文速览
开放词汇目标检测（open-set object detection）长期面临一个难题：传统检测器只能识别训练时见过的固定类别，无法根据人类的自然语言描述去定位任意物体。Grounding DINO 将强大的 Transformer 检测器 DINO 与语言-视觉多模态融合相结合，在特征增强、查询初始化、跨模态解码器三个阶段全面引入文本信息，并创新性地用"子句级文本特征"屏蔽无关类别间的注意力干扰，再通过在目标检测、图文定位、图像描述等大规模数据上进行预训练来泛化到未见类别。最终，Grounding DINO 在 COCO 零样本检测上达到 52.5 AP，在 ODinW 零样本基准上以 26.1 平均 AP 刷新纪录，同时在指代表达理解（Referring Expression Comprehension）任务上也表现出色。这项工作证明了"全阶段紧耦合多模态融合 + 大规模定位预训练"是构建通用开放世界检测器的有效路径，为后续将检测能力与生成模型等下游应用打通奠定了基础。
原文 arXiv:2303.05499;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.05499v5