Data Distillation: Towards Omni-Supervised Learning
Ilija Radosavovic Piotr Dollár Ross Girshick Georgia Gkioxari Kaiming He Facebook AI Research (FAIR)
Abstract
We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lower-bounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging real-world data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.
中文速览
想让模型从海量无标注图片中自动学到更多知识，传统半监督方法往往被限制在已有标注集的天花板以内。研究者提出了"全监督学习"（omni-supervised learning）这一框架，并配套设计了"数据蒸馏"（data distillation）方法：先用已标注数据训练一个强模型，再把它同时应用于无标注图片的多种几何变换（如多尺度、水平翻转），将多次预测结果融合成高质量的"硬标签"，最后把这些自动生成的标注和原始人工标注合并，重新训练模型。在 COCO 人体关键点检测和目标检测任务上，数据蒸馏让 Mask R-CNN 的精度明显超越了仅用 COCO 标注数据训练的强基线，关键点检测 AP 最高提升约 2 个点，接近同等规模额外人工标注带来的收益。这项工作的价值在于：它无需修改模型结构或损失函数，只需利用现有强模型对互联网级别的无标注数据进行"自我蒸馏"，便能低成本、可扩展地突破监督数据的性能上限。
原文 arXiv:1712.04440;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1712.04440v1