Contrastive Multiview Coding
Yonglong Tian MIT CSAIL Dilip Krishnan Google Research Phillip Isola MIT CSAIL
Abstract
Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a “dog” can be seen, heard, and felt). We investigate the classic hypothesis that a powerful representation is one that models view-invariant factors. We study this hypothesis under the framework of multiview contrastive learning, where we learn a representation that aims to maximize mutual information between different views of the same scene but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. We analyze key properties of the approach that make it work, finding that the contrastive loss outperforms a popular alternative based on cross-view prediction, and that the more views we learn from, the better the resulting representation captures underlying scene semantics. Our approach achieves state-of-the-art results on image and video unsupervised learning benchmarks. Code is released at: http://github.com/HobbitLong/CMC/.
中文速览
多感官感知能力让人类能同时通过视觉、听觉、触觉等多种渠道感知同一个场景，而这些渠道所共享的信息（如"这是一只狗"）往往才是真正重要的语义信号。受此启发，研究者提出了"对比多视角编码"（Contrastive Multiview Coding，CMC）方法：把同一场景的不同感知通道（如图像的亮度通道L、色彩通道ab、深度图、光流等）视为不同"视角"，通过对比学习让模型把同一场景的不同视角拉近、不同场景的视角推远，从而学到视角无关的紧凑表征，其本质是最大化不同视角表征之间的互信息。实验发现，训练时使用的视角数量越多，学到的表征语义质量越高，且对比学习目标明显优于流行的跨视角预测方法。最终CMC在图像和视频的无监督表征学习基准上达到当时最优水平，为利用多模态自然共现信号进行无监督视觉学习提供了一套简洁且可扩展的框架。
原文 arXiv:1906.05849;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.05849v5