Segment Anything Meets Point Tracking
Frano Rajič1,313{}^{1,3}start_FLOATSUPERSCRIPT 1 , 3 end_FLOATSUPERSCRIPT Lei Ke1,212{}^{1,2}start_FLOATSUPERSCRIPT 1 , 2 end_FLOATSUPERSCRIPT Yu-Wing Tai22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Chi-Keung Tang22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Martin Danelljan11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Fisher Yu11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTETH Zürich 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTHKUST 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPTEPFL
Abstract
The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models. While click and brush interactions are both well explored in interactive image segmentation, the existing methods on videos focus on mask annotation and propagation. This paper presents SAM-PT, a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. SAM-PT leverages robust and sparse point selection and propagation techniques for mask generation. Compared to traditional object-centric mask propagation strategies, we uniquely use point propagation to exploit local structure information agnostic to object semantics. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code that integrates differ
中文速览
零样本视频分割（zero-shot video segmentation）一直面临两大难题：现有模型泛化能力差、人工标注成本高。SAM-PT 把图像分割基础模型 SAM（Segment Anything Model）和长时稀疏点追踪（long-term point tracking）结合起来，只需在第一帧上标几个稀疏点，就能自动把目标物体在整段视频里分割出来，全程无需任何视频分割数据训练。与传统依赖物体级掩码传播的方法不同，SAM-PT 用点轨迹携带局部结构信息来驱动 SAM 逐帧生成分割掩码，并通过 K-Medoids 点采样、正负点联合提示以及周期性重新初始化等设计提升鲁棒性。在 DAVIS、YouTube-VOS、BDD100K 和 UVO 等主流基准上，SAM-PT 的零样本性能比现有零样本方法最高高出 7.3 个百分点，甚至超越了部分全监督方法，证明点驱动的视频分割是一条标注高效、跨域泛化能力强的可行路径。
原文 arXiv:2307.01197;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.01197v2