Customizing General-Purpose Foundation Models for Medical Report Generation
Bang Yang Peng Cheng Laboratory, Shenzhen 518055, China ADSPLAB, School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China Asif Raza ADSPLAB, School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China Yuexian Zou ADSPLAB, School of Electronic and Computer Engineering, Peking University, Shenzhen 518055, China Tong Zhang
Abstract
Medical caption prediction which can be regarded as a task of medical report generation (MRG), requires the automatic generation of coherent and accurate captions for the given medical images. However, the scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks capable of harnessing the potential artificial general intelligence power like large language models (LLMs). In this work, we propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs), in computer vision and natural language processing with a specific focus on medical report generation. Specifically, following BLIP-2, a state-of-the-art vision-language pre-training approach, we introduce our encoder-decoder-based MRG model. This model utilizes a lightweight query Transformer to connect two FMs: the giant vision Transformer EVA-ViT-g and a bilingual LLM trained to align with human intentions (referred to as ChatGLM-6B). Furthermore, we conduct ablative experiments on the trainable components of the model to identify the crucial factors for effective transfer learning. Our findings demonstrate that
中文速览
给医学图像自动生成诊断报告一直面临标注数据稀缺的瓶颈，难以充分发挥大规模神经网络的潜力。研究者借鉴视觉语言预训练框架 BLIP-2 的思路，将通用视觉基础模型 EVA-ViT-g 与双语大语言模型（large language model，LLM）ChatGLM-6B 通过一个轻量级查询变换器（Q-Former）连接起来，再用参数高效微调方法 P-tuning 对语言模型进行适配，同时解冻视觉编码器以学习医学图像特征，从而让通用基础模型（foundation model）能胜任医学报告生成任务。在 ImageCLEF 2023 医学图像标题预测竞赛中，该方案在 13 支参赛队伍里分别取得 BERTScore 第 4 名和 ROUGE-1 第 2 名的成绩，验证了复用通用基础模型的可行性与有效性。此外，研究也揭示了微调大语言模型时普遍存在的三个隐患——问答能力的灾难性遗忘、幻觉生成以及意外的语言混用——为后续构建可信赖的医疗场景专用模型指明了改进方向。
原文 arXiv:2306.05642;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.05642v1