Abstract

We show two plots of the ImageNet linear probes results (Table 2 of the paper) in Figure 1. On the left we plot performance per layer in absolute scale. Naturally the performance of the supervised model improves with depth, while all unsupervised models degrade after conv3. From the relative plot on the right, it becomes clear that with our training scheme, we can even slightly surpass supervised performance on conv1 presumably since our model is trained with sometimes very small patches, thus receiving an emphasis on learning good low level filters. The gap between all self-supervised methods and the supervised baseline increases with depth, due to the fact that the supervised model is trained for this specific task, whereas the self-supervised models learn from a surrogate task without labels.