Causal Intervention for Weakly-Supervised Semantic Segmentation
Dong Zhang1 Hanwang Zhang2 Jinhui Tang1 Xiansheng Hua3 Qianru Sun4 1School of Computer Science and Engineering, Nanjing University of Science and Technology; 2Nanyang Technological University; 3Damo Academy, Alibaba Group; 4Singapore Management University. Corresponding author.
Abstract
We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels — the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of “horse” and “person” may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.111Code is open-sourced at: https://github.com/ZHANGDONG-NJUST/CONTA
中文速览
弱监督语义分割（Weakly-Supervised Semantic Segmentation, WSSS）只用图像级标签来训练模型，核心难题在于如何生成准确的像素级伪掩码——现有方法依赖的类激活图（CAM）常因图像中对象频繁共现（如"人"总和"马"一起出现）而产生边界模糊、前景不完整等问题。本文用因果推断的视角把这个问题说清楚了：数据集中的视觉上下文是一个"混淆因子"，它让分类模型学到了像素与标签之间的虚假相关性，而非真正的因果关系。为此，作者构建了一个结构因果模型，并提出"上下文调整"方法（Context Adjustment, CONTA），通过后门调整把干扰性的上下文偏差从分类过程中剔除，以迭代方式生成质量更高的伪掩码。实验表明，CONTA 在 PASCAL VOC 2012 和 MS-COCO 两个基准上均将多种主流 WSSS 方法提升至新的最优水平，为用因果推断解决视觉弱监督学习中的数据偏差问题提供了一套有理论保障、可即插即用的通用框架。
原文 arXiv:2009.12547;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.12547v2