Open-Vocabulary DETR with Conditional Matching
Yuhang Zang1, Wei Li1, Kaiyang Zhou1, Chen Huang2, Chen Change Loy1🖂 1S-Lab, Nanyang Technological University 2Carnegie Mellon University {{\{zang0012, wei.l, kaiyang.zhou,
Abstract
Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR—hence the name OV-DETR—which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-la
中文速览
开放词汇目标检测（open-vocabulary object detection）面临的核心难题是：传统检测器只能识别训练时见过的固定类别，遇到新类别就束手无策。OV-DETR 以端到端的 Transformer 检测器 DETR 为基础，创造性地将原本针对封闭类别集的"二分图匹配"目标改写为"条件二元匹配"——用预训练视觉语言模型 CLIP 把用户输入的类别名称或示例图片转化为嵌入向量，作为条件注入解码器，让模型学会"这个查询对应图中哪个物体"的匹配关系，而无需依赖新类别的标注标签，也彻底绕开了容易过拟合到已见类别的区域提议网络（RPN）。在 LVIS 和 COCO 两个基准上，OV-DETR 的新类别检测性能分别以 1.3 和 1.8 mAP 超越此前最优方法，成为首个基于 Transformer 的端到端开放词汇检测器。这项工作的意义在于：它让检测器真正做到"用户说什么名字或给一张示例图就能检测什么目标"，大幅拓展了检测系统在真实开放场景中的适用性。
原文 arXiv:2203.11876;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.11876v2