Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
Golnaz Ghiasi Xiuye Gu Yin Cui Tsung-Yi Lin Work done while at Google.
Abstract
We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to propose segmentation masks for possible organizations. Then it learns visual-semantic alignments by aligning each word in a caption to one or a few predicted masks. We find the mask representations are the key to support learning image segmentation from captions, making it possible to scale up the dataset and vocabulary sizes. OpenSeg significantly outperforms the recent open-vocabulary method of LSeg by +19.9 mIoU on PASCAL dataset, thanks to its scalability.
中文速览
开放词汇图像分割（open-vocabulary image segmentation）长期面临一个困境：像CLIP这样靠海量图文对训练的大模型虽然能识别任意概念，却只能给整张图打标签，无法精确划定每个概念在哪里。OpenSeg的核心思路是在视觉语义对齐之前先做"视觉分组"——模型先用交叉注意力机制自动生成一批与类别无关的候选分割掩码，再把图像描述（caption）中的名词和形容词与这些掩码逐一对齐，从而用廉价的图文标注替代昂贵的逐像素标注来完成训练。由于不再依赖像素级人工标注，OpenSeg可以轻松扩大训练数据规模和词汇范围，在PASCAL数据集上比此前最强的开放词汇分割方法LSeg高出19.9个mIoU，也在更大词汇量的ADE-20k和PASCAL-Context基准上全面领先。这项工作的意义在于：它打通了"用图文描述学会分割任意概念"的完整链路，为构建真正可泛化、可扩展的视觉感知系统提供了一条切实可行的路径。
原文 arXiv:2112.12143;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.12143v2