Self-Supervised Learning of Pretext-Invariant Representations
Ishan Misra Laurens van der Maaten Facebook AI Research
Abstract
The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as “pearl”) that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised learning of image representations with good invariance properties.
中文速览
自监督学习（self-supervised learning）长期面临一个核心矛盾：现有预训练任务（pretext task）会让模型学到对图像变换"协变"的表示，也就是说表示里混入了与语义无关的变换信息，而真正有用的语义表示应当对这些变换保持"不变"。为此，论文提出了预训练不变性表示学习方法 PIRL（Pretext-Invariant Representation Learning），其核心思路是：用噪声对比估计（Noise Contrastive Estimation，NCE）损失让原图与其经过"拼图打乱"等变换后的版本产生相似的表示，同时让不同图像的表示彼此拉开距离，从而把语义信息留下、把变换信息丢掉。在 ImageNet 无监督预训练后迁移到多个下游任务的实验中，PIRL 全面超越了此前所有自监督方法，在 Pascal VOC 目标检测任务上甚至比有监督预训练还要好，AP 指标高出约 5 个百分点。这项工作说明，只需在训练目标上做一个关键改变——从"预测变换"转向"消除变换影响"——就能大幅提升自监督表示的语义质量，为无标注图像预训练提供了一条切实可行的新路径。
原文 arXiv:1912.01991;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.01991v1