Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Xiuye Gu1, Tsung-Yi Lin2, Weicheng Kuo1, Yin Cui1 1Google Research, 2Nvidia {xiuyegu, weicheng, Work done while Xiuye was a Google AI Resident and Tsung-Yi was at Google.
Abstract
We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask APr with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 APr. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP50 on PASCAL VOC, 36.6 AP on COCO an
中文速览
用海量图文对预训练的模型（如CLIP/ALIGN）虽然能识别任意类别，却无法定位图中的具体目标，而传统目标检测器又受限于标注类别无法泛化到新类别——ViLD正是为了打通这两者而提出的。研究者设计了一种知识蒸馏训练框架，让预训练的图文分类模型作为"教师"，一方面用其文本编码器生成类别嵌入来替换检测器的分类头（ViLD-text），另一方面用其图像编码器对候选框区域编码、并让检测器的区域特征向这些嵌入对齐（ViLD-image），从而把开放词汇的识别能力"蒸馏"进一个标准的两阶段检测器（Mask R-CNN）中。在LVIS基准上，ViLD对从未见过的稀有类别取得16.1 APr，甚至比全监督方法高出3.8个点；换用更强的ALIGN教师模型后提升至26.3 APr，并可直接迁移到PASCAL VOC、COCO等数据集而无需微调。这项工作的意义在于，它首次在千余类别规模上验证了"无需为新类别收集检测标注、仅靠文本描述就能检测任意目标"的可行性，为低成本扩展检测词汇表提供了切实可行的路径。
原文 arXiv:2104.13921;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.13921v3