Self-training Avoids Using Spurious Features Under Domain Shift
Yining Chen∗, Colin Wei∗, Ananya Kumar, Tengyu Ma Department of Computer Science Stanford University {cynnjjs, colinwei, ananya1, Equal Contribution
Abstract
In unsupervised domain adaptation, existing theory focuses on situations where the source and target domains are close. In practice, conditional entropy minimization and pseudo-labeling work even when the domain shifts are much larger than those analyzed by existing theory. We identify and analyze one particular setting where the domain shift can be large, but these algorithms provably work: certain spurious features correlate with the label in the source domain but are independent of the label in the target. Our analysis considers linear classification where the spurious features are Gaussian and the non-spurious features are a mixture of log-concave distributions. For this setting, we prove that entropy minimization on unlabeled target data will avoid using the spurious feature if initialized with a decently accurate source classifier, even though the objective is non-convex and contains multiple bad local minima using the spurious features. We verify our theory for spurious domain shift tasks on semi-synthetic Celeb-A and MNIST datasets. Our results suggest that practitioners collect and self-train on large, diverse datasets to reduce biases in classifiers even if labeling is im
中文速览
在现实机器学习中，训练数据常含"虚假关联"（spurious correlation）——某些特征在训练集里与标签相关，但在真实部署环境中却与标签无关，导致模型泛化失败。针对这一问题，研究者从理论上证明了一件看似出乎意料的事：即便源域和目标域之间的分布偏移很大，只要用一个在源域上训练得还算准确的分类器作为初始化，在无标签目标域数据上做自训练（self-training）或条件熵最小化（conditional entropy minimization），模型就能自动"忘掉"那些虚假特征、只保留真正有用的信号特征。理论分析针对线性分类器展开，其中虚假特征服从高斯分布、信号特征服从对数凹混合分布，主要定理给出了有限样本下的收敛保证，并在半合成的Colored MNIST和CelebA数据集上用多层神经网络验证了理论结论。这一结果的实践意义在于：即使为海量数据打标签不现实，收集大规模、多样化的无标签数据并进行自训练，也能有效减少模型对数据偏差的依赖，从而提升公平性和鲁棒性。
原文 arXiv:2006.10032;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.10032v3