In Search of Lost Domain Generalization
Ishaan Gulrajani and David Lopez-Paz Facebook AI Research Alphabetical order, equal contribution.
Abstract
The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions—datasets, architectures, and model selection criteria—render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain gene
中文速览
机器学习模型在训练分布之外往往会表现失常，领域泛化（domain generalization）算法正是为了解决这一问题——让模型在从未见过的数据分布上也能预测准确。然而现有大量算法的实验条件各不相同，数据集、网络架构、超参选择标准五花八门，根本无法公平比较。为此，作者构建了一个名为 DomainBed 的标准化测试平台，涵盖七个多域数据集、九种基线算法和三种模型选择方案，在严格统一的条件下对各算法进行系统评测。结果令人意外：只要配合现代神经网络架构和数据增强技术，最朴素的经验风险最小化（empirical risk minimization, ERM）在所有数据集上均达到了当前最优水平，专门设计的领域泛化算法并未带来稳定的性能提升。这一发现促使研究者重新审视该领域的真实进展，而 DomainBed 的公开发布也为后续可复现、可比较的研究提供了坚实基础。
原文 arXiv:2007.01434;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.01434v1