High-Quality Entity Segmentation
Lu Qi1, Jason Kuen2∗, Weidong Guo3∗, Tiancheng Shen4∗, Jiuxiang Gu2, Jiaya Jia4, Zhe Lin2, Ming-Hsuan Yang1, 1The University of California, Merced 2Adobe Research 3QQ Brower Lab, Tencent, 4The Chinese University of Hong Kong Equal contribution.
Abstract
Dense image segmentation tasks (e.g., semantic, panoptic) are useful for image editing, but existing methods can hardly generalize well in an in-the-wild setting where there are unrestricted image domains, classes, and image resolution & quality variations. Motivated by these observations, we construct a new entity segmentation dataset, with a strong focus on high-quality dense segmentation in the wild. The dataset contains images spanning diverse image domains and entities, along with plentiful high-resolution images and high-quality mask annotations for training and testing. Given the high-quality and -resolution nature of the dataset, we propose CropFormer which is designed to tackle the intractability of instance-level segmentation on high-resolution images. It improves mask prediction by fusing high-res image crops that provides more fine-grained image details and the full image. CropFormer is the first query-based Transformer architecture that can effectively fuse mask predictions from multiple image views, by learning queries that effectively associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the chal
中文速览
现有的图像分割方法在面对"野外"场景时泛化能力差，既难以处理高分辨率图像，又受限于封闭类别集，无法满足图像编辑等实际需求。为此，研究者构建了EntitySeg数据集——一个包含33,227张图片的大规模开放世界分割数据集，超过70%的图片分辨率高于2000px，且所有标注均以类别无关的方式对每个语义连贯区域进行精细的像素级标注。针对高分辨率图像分割计算量大、现有Transformer方法难以融合多视角预测的难题，作者提出了CropFormer模型：它通过可学习的查询向量动态关联完整图像与其高分辨率局部裁剪（crop）中的同一实体，在获取全局上下文的同时利用局部细节，实现了多视角掩码预测的有效融合，在实体分割任务上取得了1.9 AP的显著提升，并在传统分割任务上也持续改善了准确率。这项工作填补了高质量开放世界分割数据集的空白，为图像编辑等需要精细分割的应用场景提供了重要的数据与方法基础。
原文 arXiv:2211.05776;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.05776v3