Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
Zefan Li14 Work done while visiting Johns Hopkins University. Chenxi Liu2 Now at Waymo. Alan Yuille2 Bingbing Ni14 Corresponding author. Wenjun Zhang1 Wen Gao3 1Shanghai Jiao Tong University 2Johns Hopkins University 3Peking University 4MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University
Abstract
Unsupervised learning methods have recently shown their competitiveness against supervised training. Typically, these methods use a single objective to train the entire network. But one distinct advantage of unsupervised over supervised learning is that the former possesses more variety and freedom in designing the objective. In this work, we explore new dimensions of unsupervised learning by proposing the Progressive Stage-wise Learning (PSL) framework. For a given unsupervised task, we design multi-level tasks and define different learning stages for the deep network. Early learning stages are forced to focus on low-level tasks while late stages are guided to extract deeper information through harder tasks. We discover that by progressive stage-wise learning, unsupervised feature representation can be effectively enhanced. Our extensive experiments show that PSL consistently improves results for the leading unsupervised learning methods.
中文速览
无监督视觉表征学习长期面临如何让深层网络从易到难、循序渐进地提取特征这一难题。本文提出了渐进式分阶段学习框架（Progressive Stage-wise Learning，PSL）：针对任意无监督任务（如拼图、旋转预测或对比学习），先将其拆分为由简到难的多级子任务，再把神经网络按层深切分为早、中、晚三个阶段，让浅层专注简单任务、深层挑战更难任务，每个阶段的梯度只在本阶段内回传，避免了长路径带来的梯度误差积累。实验结果表明，PSL作为即插即用模块，能持续提升拼图预测、图像旋转和SimCLR等主流无监督方法在线性分类、半监督学习和迁移学习上的性能。这项工作的意义在于，它为无监督学习提供了一个与具体任务设计正交的新优化维度，只需在现有方法上叠加PSL即可获得稳定增益，具有较强的通用性和实用价值。
原文 arXiv:2106.05554;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.05554v2