Context Encoders: Feature Learning by Inpainting
Deepak Pathak Philipp Krähenbühl Jeff Donahue Trevor Darrell Alexei A. Efros University of California, Berkeley
Abstract
We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders – a convolutional neural network trained to generate the contents of an arbitrary image region conditioned on its surroundings. In order to succeed at this task, context encoders need to both understand the content of the entire image, as well as produce a plausible hypothesis for the missing part(s). When training context encoders, we have experimented with both a standard pixel-wise reconstruction loss, as well as a reconstruction plus an adversarial loss. The latter produces much sharper results because it can better handle multiple modes in the output. We found that a context encoder learns a representation that captures not just appearance but also the semantics of visual structures. We quantitatively demonstrate the effectiveness of our learned features for CNN pre-training on classification, detection, and segmentation tasks. Furthermore, context encoders can be used for semantic inpainting tasks, either stand-alone or as initialization for non-parametric methods.
中文速览
人脑能轻松脑补出一张图里被遮住的部分，但让计算机做到这一点却很难——这篇文章正是要解决这个问题：如何让神经网络在没有任何人工标注的情况下，自动学到有语义意义的视觉特征。作者提出了"上下文编码器"（Context Encoder），这是一种编码器-解码器结构的卷积神经网络，训练方式是：把图像中一大块区域抠掉，让网络只看周围的像素去预测缺失部分的内容，同时结合像素级重建损失（L2 loss）和生成对抗损失（adversarial loss）来避免预测结果模糊。实验表明，这种方法不仅能生成视觉上合理的图像补全结果（语义修复/inpainting），学到的特征表示还能迁移到图像分类、目标检测、语义分割等任务上，与当时最先进的无监督方法旗鼓相当。它的重要性在于：无需任何标注数据，仅靠"预测被遮挡内容"这一自监督任务，就能驱动模型理解图像的深层语义结构，为后续大量自监督视觉表示学习工作奠定了重要基础。
原文 arXiv:1604.07379;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1604.07379v2