Breaking Inter-Layer Co-Adaptation by Classifier Anonymization
Ikuro Sato Kohta Ishikawa Guoqing Liu Masayuki Tanaka
Abstract
This study addresses an issue of co-adaptation between a feature extractor and a classifier in a neural network. A naïve joint optimization of a feature extractor and a classifier often brings situations in which an excessively complex feature distribution adapted to a very specific classifier degrades the test performance. We introduce a method called Feature-extractor Optimization through Classifier Anonymization (FOCA), which is designed to avoid an explicit co-adaptation between a feature extractor and a particular classifier by using many randomly-generated, weak classifiers during optimization. We put forth a mathematical proposition that states the FOCA features form a point-like distribution within the same class in a class-separable fashion under special conditions. Real-data experiments under more general conditions provide supportive evidences.
中文速览
神经网络在联合训练特征提取器和分类器时，两者会彼此"绑架"——特征提取器学出专门迁就某个分类器的超复杂特征分布，一旦换个分类器或迁移到新任务，性能就大幅下降。为此，作者提出了一种叫做"分类器匿名化特征优化"（FOCA，Feature-extractor Optimization through Classifier Anonymization）的方法：在训练特征提取器时，不使用固定的分类器，而是每次随机生成只见过一小批数据的弱分类器，让特征提取器必须同时取悦大量不同的弱分类器，从而无法和任何单一分类器形成强耦合。理论分析表明，在特定条件下FOCA会驱使同类样本的特征收缩为同一个点、不同类样本的特征则相互分离，即形成极为简洁的类内点状分布；在真实数据实验中也观察到类似现象，并验证了FOCA特征在小样本、跨分类器、跨域迁移等场景下的强鲁棒性。这一工作为从根本上消除神经网络层间共适应问题提供了新思路，对迁移学习和模型泛化具有实际意义。
原文 arXiv:1906.01150;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.01150v1