LViT: Language meets Vision Transformer in Medical Image Segmentation
Zihan Li, Yunxiang Li, Qingde Li, Puyang Wang, Dazhou Guo, Le Lu, , Dakai Jin, , You Zhang, Qingqi Hong Zihan Li is with Xiamen University and the Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA (e-mail: Li and You Zhang are with the Department of Radiation Oncology, UT Southwestern Medical Center, Dallas, TX 75235, USA.Qingde Li is with the Department of Computer Science, University of Hull, Hull, HU6 7RX, UK.Puyang Wang is with DAMO Academy, Alibaba Group, Hangzhou 310024, China.Dazhou Guo, Le Lu, and Dakai Jin are with DAMO Academy, Alibaba Group, New York, NY 10014, USA.Qingqi Hong is with Xiamen University, Xiamen 361005, China. (e-mail: author: Qingqi Hong
Abstract
Deep learning has been widely used in medical image segmentation and other aspects. However, the performance of existing medical image segmentation models has been limited by the challenge of obtaining sufficient high-quality labeled data due to the prohibitive data annotation cost. To alleviate this limitation, we propose a new text-augmented medical image segmentation model LViT (Language meets Vision Transformer). In our LViT model, medical text annotation is incorporated to compensate for the quality deficiency in image data. In addition, the text information can guide to generate pseudo labels of improved quality in the semi-supervised learning. We also propose an Exponential Pseudo label Iteration mechanism (EPI) to help the Pixel-Level Attention Module (PLAM) preserve local image features in semi-supervised LViT setting. In our model, LV (Language-Vision) loss is designed to supervise the training of unlabeled images using text information directly. For evaluation, we construct three multimodal medical segmentation datasets (image + text) containing X-rays and CT images. Experimental results show that our proposed LViT has superior segmentation performance in both fully-supe
中文速览
医学图像分割因标注成本高昂、高质量标签数据稀缺而长期受限，而随诊断报告天然生成的医学文本却几乎零成本可得。针对这一痛点，研究者提出了 LViT（Language meets Vision Transformer）模型，将医学文本注释融入图像分割流程：模型采用 CNN 与 Transformer 双 U 形分支结构，用轻量级嵌入层提取文本特征并与图像特征融合，同时设计了像素级注意力模块（PLAM）来保留图像局部细节；在半监督场景下，进一步引入指数伪标签迭代机制（EPI）和语言-视觉（LV）损失，让文本信息直接监督无标签图像的训练并持续提升伪标签质量。在三个自建多模态医学分割数据集（X 光与 CT）上的实验表明，LViT 在全监督和半监督设置下均优于现有方法，尤其值得关注的是，仅使用 1/4 标签数据时其性能即可媲美完全监督方法。该工作证明了"免费"的医学文本是提升分割精度、缓解标注瓶颈的有效补充信号，对数据稀缺的医疗 AI 场景具有重要实用价值。
原文 arXiv:2206.14718;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.14718v4