Cross-Validation, Risk Estimation, and Model Selection
Stefan Wager Stanford University
Abstract
Cross-validation is a popular non-parametric method for evaluating the accuracy of a predictive rule. The usefulness of cross-validation depends on the task we want to employ it for. In this note, I discuss a simple non-parametric setting, and find that cross-validation is asymptotically uninformative about the expected test error of any given predictive rule, but allows for asymptotically consistent model selection. The reason for this phenomenon is that the leading-order error term of cross-validation doesn’t depend on the model being evaluated, and so cancels out when we compare two models. This note was prepared as a comment on a paper by Rosset and Tibshirani, forthcoming in the Journal of the American Statistical Association.
中文速览
交叉验证（cross-validation）既被用来估计预测模型的绝对误差，也被用来在多个模型之间做选择，但这两个任务的难度其实截然不同。作者在随机协变量的渐近框架下严格分析了K折交叉验证的行为，发现交叉验证估计量的主导误差项只取决于最优预测函数本身、与被评估的具体模型无关，导致交叉验证对单个模型的测试误差渐近上毫无信息量——用一个不依赖任何模型的常数估计反而更准。然而，一旦拿两个模型做比较，这个共同的噪声项会在作差时完全抵消，使得交叉验证能够渐近一致地选出更准的那个模型。这一结论提醒从业者：看交叉验证的绝对误差数值意义有限，而用它做模型筛选却是可靠的，理解这一区别对正确解读机器学习实验结果至关重要。
原文 arXiv:1909.11696;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.11696v1