PromptDet: Towards Open-vocabulary Detection using Uncurated Images
Chengjian Feng 11 Yujie Zhong 11 Zequn Jie 11 Xiangxiang Chu Haibing Ren 1111 Xiaolin Wei 11 Weidi Xie 22 ✉ ✉ Lin Ma 11
Abstract
The goal of this work is to establish a scalable pipeline for expanding an object detector towards novel/unseen categories, using zero manual annotations. To achieve that, we make the following four contributions: (i) in pursuit of generalisation, we propose a two-stage open-vocabulary object detector, where the class-agnostic object proposals are classified with a text encoder from pre-trained visual-language model; (ii) To pair the visual latent space (of RPN box proposals) with that of the pre-trained text encoder, we propose the idea of regional prompt learning to align the textual embedding space with regional visual object features; (iii) To scale up the learning procedure towards detecting a wider spectrum of objects, we exploit the available online resource via a novel self-training framework, which allows to train the proposed detector on a large corpus of noisy uncurated web images. Lastly, (iv) to evaluate our proposed detector, termed as PromptDet, we conduct extensive experiments on the challenging LVIS and MS-COCO dataset. PromptDet shows superior performance over existing approaches with fewer additional training images and zero manual annotations whatsoever. Project
中文速览
开放词汇目标检测（open-vocabulary object detection）的核心难题在于：模型训练时只见过少数"基础类别"，推理时却要识别任意新类别，而人工标注代价极高。PromptDet 的解法是把预训练视觉-语言模型 CLIP 的文本编码器直接当作分类器，让区域候选框（RPN proposals）的视觉特征与文字描述在同一空间里对齐；为了弥补"区域级视觉特征"与"场景级文本嵌入"之间的天然偏差，作者提出区域提示学习（Regional Prompt Learning，RPL），在文本输入前后插入可学习的连续向量，把文本潜空间"掰向"更贴近目标区域的方向。更进一步，为了让检测器覆盖更广泛的类别，方法还引入了一套自训练框架：用更新后的提示词从网络上自动检索对应类别的图片，再用生成的伪标签迭代训练检测器，全程无需任何人工标注。在 LVIS 和 MS-COCO 基准上，PromptDet 以更少的训练图片和更低的计算开销超越了此前所有方法，证明了"提示学习+网络图片自训练"是一条低成本、可扩展地扩充检测词汇的有效路径。
原文 arXiv:2203.16513;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.16513v2