Learning unbiased features
Yujia Li University of Toronto Kevin Swersky University of Toronto Richard Zemel University of Toronto Canadian Institute for Advanced Research
Abstract
A key element in transfer learning is representation learning; if representations can be developed that expose the relevant factors underlying the data, then new tasks and domains can be learned readily based on mappings of these salient factors. We propose that an important aim for these representations are to be unbiased. Different forms of representation learning can be derived from alternative definitions of unwanted bias, e.g., bias to particular tasks, domains, or irrelevant underlying data dimensions. One very useful approach to estimating the amount of bias in a representation comes from maximum mean discrepancy (MMD) [5], a measure of distance between probability distributions. We are not the first to suggest that MMD can be a useful criterion in developing representations that apply across multiple domains or tasks [1]. However, in this paper we describe a number of novel applications of this criterion that we have devised, all based on the idea of developing unbiased representations. These formulations include: a standard domain adaptation framework; a method of learning invariant representations; an approach based on noise-insensitive autoencoders; and a novel form of g
中文速览
迁移学习的核心难题在于如何学到跨域、跨任务都好用的无偏表示，现有方法要么依赖线性变换、要么目标函数不一致。本文统一用最大均值差异（Maximum Mean Discrepancy, MMD）这一分布距离度量作为正则化惩罚项，将其嵌入深度神经网络，衍生出四种应用：有标签的跨域迁移（领域自适应）、多光照条件下的人脸身份识别（不变特征学习）、抗噪声自编码器，以及用单一一致目标替代复杂对抗博弈的生成模型训练。在亚马逊评论情感分类、耶鲁人脸等基准上，加入MMD惩罚的神经网络均显著优于SVM、普通神经网络及TCA等基线，人脸识别准确率从72%提升到82%，生成模型也能产出视觉上可信的MNIST样本。这项工作的重要性在于，它用一个简洁统一的数学工具串联起监督与无监督迁移学习的多个重要场景，为大规模、标注稀疏数据下的深度表示学习提供了实用且理论清晰的框架。
原文 arXiv:1412.5244;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.5244v1