Open World Entity Segmentation
Lu Qi, Jason Kuen†, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, Jiaya Jia L .Qi and J. Jia are with the Department of CSE, The Chinese University of Hong Kong. Jason Kuen, J .Gu and Z .Lin are with the Adobe Research. Y .Wang is with the Shanghai AI Lab. H. Zhao is with the Department of Computer Science at The University of Hong Kong. Philip Torr is with the Department of Engineering Science, University of Oxford. Jason Kuen† is the corresponding author. The contact emails are (Lu Qi) and (Jason Kuen).
Abstract
We introduce a new image segmentation task, called Entity Segmentation (ES), which aims to segment all visual entities (objects and stuffs) in an image without predicting their semantic labels. By removing the need of class label prediction, the models trained for such task can focus more on improving segmentation quality. It has many practical applications such as image manipulation and editing where the quality of segmentation masks is crucial but class labels are less important. We conduct the first-ever study to investigate the feasibility of convolutional center-based representation to segment things and stuffs in a unified manner, and show that such representation fits exceptionally well in the context of ES. More specifically, we propose a CondInst-like fully-convolutional architecture with two novel modules specifically designed to exploit the class-agnostic and non-overlapping requirements of ES. Experiments show that the models designed and trained for ES significantly outperforms popular class-specific panoptic segmentation models in terms of segmentation quality. Moreover, an ES model can be easily trained on a combination of multiple datasets without the need to resolv
中文速览
图像分割领域长期以来要求模型在切割出每个区域的同时还必须给它贴上类别标签，这既容易造成类别混淆，又难以泛化到训练集以外的新类别。研究者提出了"实体分割"（Entity Segmentation，ES）这一新任务——只要把图像中所有视觉实体（物体和背景区域）的轮廓切干净，完全不预测类别，从而把模型的全部能力集中在分割质量本身。为此，他们设计了一个基于卷积中心点表示的全卷积框架（受CondInst启发），并引入"全局核库模块"和"重叠抑制模块"两个新组件，分别利用实体间共享的纹理/边缘特征、以及强制预测掩码互不重叠。实验表明，该方法在分割质量上显著超越主流全景分割模型，还能轻松融合多个数据集联合训练而无需解决标签冲突，在COCO上训练的模型直接迁移到ADE20K等未见域也表现出色。这项工作为图像编辑、抠图等对分割精度要求高但不关心类别的应用场景提供了更合适的范式，同时也为开放世界图像分割开辟了一条新路径。
原文 arXiv:2107.14228;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2107.14228v3