A Simple Framework for Open-Vocabulary Segmentation and Detection
Hao Zhang1,2, Feng Li1,2∗, Xueyan Zou4, Shilong Liu2,5, Chunyuan Li3 Jianfeng Gao3, Jianwei Yang3, Lei Zhang2† 1The Hong Kong University of Science and Technology. 2International Digital Economy Academy (IDEA). 3Microsoft Research at Redmond. 4University of Wisconsin-Madison. 5Dept. of CST., BNRist Center, Institute for AI, Tsinghua University. {hzhangcx, Equal contribution. List in random.Equal advisory contribution.
Abstract
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$ ) task discrepancy – segmentation requires extracting masks for both foreground objects and background stuff, while detection merely cares about the former; $ii$ ) data discrepancy – box and mask annotations are with different spatial granularity, and thus not directly interchangeable. To address these issues, we propose a decoupled decoding to reduce the interference between foreground/background and a conditioned mask decoding to assist in generating masks for given boxes. To this end, we develop a simple encoder-decoder model encompassing all three techniques and train it jointly on COCO and ††footnotetext: This work is developed during an internship at IDEA. Objects365. A
中文速览
同一个神经网络同时学好目标检测和图像分割，长期以来是个难题，根源在于两类任务的标注词表大小不同、标注粒度（边框 vs. 掩码）也不同。OpenSeeD 用一个共享文本编码器把检测数据集 Objects365 和分割数据集 COCO 的所有概念映射到同一语义空间，再通过两项关键设计弥合两类任务的内在差距：一是把解码器的查询向量拆分为前景查询和背景查询，避免只关注前景的检测任务干扰需要同时识别背景的分割任务；二是引入条件掩码解码，让模型从检测数据的边框中辅助生成掩码，反过来也帮助分割头从更大规模的检测数据中受益。实验结果显示，OpenSeeD 在开放词表实例分割与全景分割的5个数据集上均超越此前最优方法，在 LVIS 和 ODinW 上的开放词表检测也优于同类工作，迁移到具体任务后在 COCO、ADE20K、Cityscapes 等数据集上再次刷新多项纪录。这项工作的意义在于，它首次证明检测与分割可以在单一模型中联合训练并互相增益，为开放世界通用感知系统提供了一个坚实的基线。
原文 arXiv:2303.08131;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.08131v3