Confident Learning: Estimating Uncertainty in Dataset Labels
\nameCurtis G. Northcutt \addrCleanlab \addrMassachusetts Institute of Technology, Department of EECS, Cambridge, MA, USA \AND\nameLu Jiang \addrGoogle Research, Mountain View, CA, USA \AND\nameIsaac L. Chuang \addrMassachusetts Institute of Technology, Department of EECS, Department of Physics, Cambridge, MA, USA
Abstract
Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is a data-centric approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amaz
中文速览
数据集里混入错误标签是机器学习的常见难题，但大多数方法只顾改模型或损失函数，从不正面回答"哪些样本被贴错了标签"。置信学习（Confident Learning，CL）反其道而行之，直接估计带噪标签与真实标签之间的联合分布，从而系统地定位并剔除标注错误。具体做法是：先用任意分类器做交叉验证得到样本的类别预测概率，再通过"置信联合"矩阵统计每对（噪声类别，真实类别）组合的样本数量，最后按此分布过滤掉可疑错误样本并重新训练。理论上，文章给出了CL精确识别标签错误的充分条件，并证明其估计一致性；实验上，CL在CIFAR噪声标签学习任务中超越七种竞争方法，在ImageNet上定量发现了类别重叠与标注混乱（如645张导弹图片被误标为其父类"抛射物"），还成功揭示了MNIST数据集中长期被忽视的标注错误。这项工作的重要意义在于，它将"数据质量"置于核心，提供了一套与模型、数据类型无关的通用数据清洗框架，并以开源工具cleanlab的形式落地，为大规模数据集的标注审查提供了可复现的实用方案。
原文 arXiv:1911.00068;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.00068v6