All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning
Can Cui Computer Science Vanderbilt University Nashville, USA Ruining Deng Computer Science Vanderbilt University Nashville, USA Quan Liu Computer Science Vanderbilt University Nashville, USA Tianyuan Yao Computer Science Vanderbilt University Nashville, USA Shunxing Bao Electrical and Computer Engineering Vanderbilt University Nashville, USA Lucas W. Remedios Computer Science Vanderbilt University Nashville, USA Bennett A. Landman Electrical and Computer Engineering Vanderbilt University Nashville, USA Yucheng Tang NVIDIA Cooperation Redmond, WA, USA Yuankai Huo Computer Science Vanderbilt University Nashville, USA
Abstract
The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competi
中文速览
细胞核分割（nuclei segmentation）是病理图像分析的核心任务，但逐像素手工标注极其耗时费力。作者提出了一套名为 All-in-SAM 的流程，将通用分割基础模型 SAM（Segment Anything Model）贯穿于整个开发链路：先用 SAM 配合少量弱标注（如边界框）自动生成像素级伪标签，再用这些伪标签对 SAM 进行微调，推理时完全不需要人工输入提示词。在公开数据集 Monuseg 上的实验表明，该方法不仅在完整标注场景下超越了 nnUNet、LViT 等当前最优方法，在仅使用 0.5% 训练数据或仅有弱标注的极端条件下依然保持竞争力，而其他方法的性能则大幅下滑。这项工作的价值在于，它为医学图像分割提供了一条"低标注成本、免推理提示"的实用路径，有望大幅降低临床场景中的人工标注负担。
原文 arXiv:2307.00290;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.00290v2