Glister: Generalization based Data Subset Selection for Efficient and Robust Learning
Krishnateja Killamsetty1, Durga Sivasubramanian 2, Ganesh Ramakrishnan 2, Rishabh Iyer1,2
Abstract
Large scale machine learning and deep models are extremely data-hungry. Unfortunately, obtaining large amounts of labeled data is expensive, and training state-of-the-art models (with hyperparameter tuning) requires significant computing resources and time. Secondly, real-world data is noisy and imbalanced. As a result, several recent papers try to make the training process more efficient and robust. However, most existing work either focuses on robustness or efficiency, but not both. In this work, we introduce Glister, a GeneraLIzation based data Subset selecTion for Efficient and Robust learning framework. We formulate Glister as a mixed discrete-continuous bi-level optimization problem to select a subset of the training data, which maximizes the log-likelihood on a held-out validation set. We then analyze Glister for simple classifiers such as gaussian and multinomial naive-bayes, k-nearest neighbor classifier, and linear regression and show connections to submodularity. Next, we propose an iterative online algorithm Glister-Online, which performs data selection iteratively along with the parameter updates and can be applied to any loss-based learning algorithm. We then show tha
中文速览
训练大规模机器学习和深度学习模型面临两个痛点：一是标注数据成本高、训练耗时长；二是真实数据往往含有噪声标签和类别不平衡，导致模型鲁棒性差。GLISTER（基于泛化的数据子集选择框架）通过将数据选择建模为一个双层优化问题，从训练集中挑出一个小子集，使模型在独立验证集上的对数似然最大化，从而同时追求效率与鲁棒性。具体实现上，提出了迭代在线算法 GLISTER-Online，在模型参数更新的同时周期性地刷新所选子集，并证明对交叉熵、铰链损失等常见损失函数，内层离散选择问题具有（弱）子模性，保证了算法的收敛性；此外还扩展出 GLISTER-Active 用于批量主动学习。实验表明，GLISTER 在加速训练（3×–6× 提速）、噪声/不平衡场景下的鲁棒学习以及主动学习三类任务上均优于或持平于现有最优方法，且计算开销更低，是首个在效率和鲁棒性上同时取得实质性进展的统一框架。
原文 arXiv:2012.10630;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.10630v4