Learning to Exploit Temporal Structure for Biomedical Vision–Language Processing
Shruthi Bannur, Stephanie Hyland∗*, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori Javier Alvarez-Valle, and Ozan Oktay Microsoft Health Futures These authors contributed equally.Corresponding author:
Abstract
Self-supervised learning in vision–language processing (VLP) exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN–Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art (SOTA) performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quan
中文速览
临床放射科报告常常提及"与上次相比病灶有所改善"这类描述，但过去的视觉-语言预训练（Vision-Language Pre-training, VLP）模型只把单张图像和对应报告配对训练，完全忽略了历史影像，导致文本与图像之间存在语义歧义，也浪费了数据中天然存在的时序监督信号。为解决这一问题，研究者提出了 BioViL-T 框架，其核心是一个 CNN-Transformer 混合多图像编码器，能够同时接收当前胸片和历史胸片，通过自注意力机制在图像区域之间跨时间点建立对应关系，无需显式图像配准，并在历史图像缺失时自动退化为单图模式；在此基础上，文本编码器借助对比学习和图像引导的掩码语言建模与图像编码器联合训练。实验表明，BioViL-T 在病变进展分类、短语定位、报告生成等多个下游任务上均达到当前最优水平，甚至在不涉及时序的肺炎检测等静态任务上也有稳定提升，同时研究团队还发布了由放射科专家标注的多模态时序基准数据集 MS-CXR-T。这项工作说明，合理利用医学影像数据中本就存在的时序结构，无需额外采集数据就能显著提升视觉-语言模型的表示质量，对推动临床 AI 的实际落地具有重要意义。
原文 arXiv:2301.04558;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.04558v2