Marginalizing Corrupted Features
\nameLaurens van der Maaten \addrPattern Recognition and Bioinformatics Group Delft University of Technology Mekelweg 4, 2628 CD Delft, The Netherlands \AND\nameMinmin Chen111Most of this work was performed while the author was at Washington University in St. Louis. \addrCriteo Palo Alto, CA 94301, USA \AND\nameStephen Tyree \nameKilian Q. Weinberger \addrDepartment of Computer Science and Engineering Washington University in St. Louis St. Louis, MO 63130, USA
Abstract
The goal of machine learning is to develop predictors that generalize well to test data. Ideally, this is achieved by training on an almost infinitely large training data set that captures all variations in the data distribution. In practical learning settings, however, we do not have infinite data and our predictors may overfit. Overfitting may be combatted, for example, by adding a regularizer to the training objective or by defining a prior over the model parameters and performing Bayesian inference. In this paper, we propose a third, alternative approach to combat overfitting: we extend the training set with infinitely many artificial training examples that are obtained by corrupting the original training data. We show that this approach is practical and efficient for a range of predictors and corruption models. Our approach, called marginalized corrupted features (MCF), trains robust predictors by minimizing the expected value of the loss function under the corruption model. We show empirically on a variety of data sets that MCF classifiers can be trained efficiently, may generalize substantially better to test data, and are also more robust to feature deletion at test time.
中文速览
训练数据有限时，机器学习模型很容易过拟合——把噪声当规律来学。为此，研究者提出了"边缘化损坏特征"（Marginalized Corrupted Features，MCF）方法：不再手动设计正则化项或先验分布，而是对有限训练样本施加随机噪声（如高斯噪声、泊松噪声、随机特征置零），在理论上生成无穷多个扰动副本，再通过对噪声分布求期望（即边缘化）将损失函数转化为可直接优化的解析形式，训练代价仅与原始样本量成线性关系。实验表明，MCF 在文本分类、图像分类和生物信息学等多类任务上均能显著降低泛化误差，并对测试时特征缺失场景表现出更强的鲁棒性。这一方法实现简单、计算高效，可作为传统 L1/L2 正则化的有力替代方案，同时还提供了一种更直观的视角——通过描述"数据可能如何变化"而非"参数应该有多小"来抑制过拟合。
原文 arXiv:1402.7001;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1402.7001v1