CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Jeremy Irvin,1,* Pranav Rajpurkar,1,* Michael Ko,1 Yifan Yu,1 Silviana Ciurea-Ilcus,1 Chris Chute,1 Henrik Marklund,1 Behzad Haghgoo,1 Robyn Ball,2 Katie Shpanskaya,3 Jayne Seekins,3 David A. Mong,3 Safwan S. Halabi,3 Jesse K. Sandberg,3 Ricky Jones,3 David B. Larson,3 Curtis P. Langlotz,3 Bhavik N. Patel,3 Matthew P. Lungren,3,† Andrew Y. Ng1,† 1Department of Computer Science, Stanford University 2Department of Medicine, Stanford University 3Department of Radiology, Stanford University *Equal contribution †Equal contribution {jirvin16,
Abstract
Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. W
中文速览
医学影像AI长期面临的瓶颈之一是缺乏足够大、标注质量足够高的公开数据集。斯坦福团队为此构建了CheXpert——一个包含65,240名患者共224,316张胸部X光片的大规模数据集，并设计了一套自动标注工具，从放射科报告中提取14种常见病理发现，同时保留报告中固有的"不确定"标签（如"可能""不排除"等表述）。针对这些不确定标签，研究者系统比较了忽略、映射为0或1、自训练、三分类等多种训练策略，发现不同策略对不同病理的效果各有优劣。最终模型在心脏肥大、肺水肿、胸腔积液三项病变的检测上，其ROC曲线和PR曲线全面超越3位额外参与测试的执业放射科医生的表现。这项工作的价值不仅在于提供了迄今最大的公开胸片基准数据集，更在于系统梳理了标注不确定性这一真实临床场景中普遍存在却常被忽视的问题，为后续模型开发提供了可复现的评测标准。
原文 arXiv:1901.07031;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.07031v1