The Phase Transition for the Existence of the Maximum Likelihood Estimate in High-dimensional Logistic Regression
Emmanuel J. Candès Department of Statistics, Stanford University, Stanford, CA 94305, U.S.A.Department of Mathematics, Stanford University, Stanford, CA 94305, U.S.A. Pragya Sur
Abstract
This paper rigorously establishes that the existence of the maximum likelihood estimate (MLE) in high-dimensional logistic regression models with Gaussian covariates undergoes a sharp ‘phase transition’. We introduce an explicit boundary curve $h_{\text{MLE}}$ , parameterized by two scalars measuring the overall magnitude of the unknown sequence of regression coefficients, with the following property: in the limit of large sample sizes $n$ and number of features $p$ proportioned in such a way that $p/n\rightarrow\kappa$ , we show that if the problem is sufficiently high dimensional in the sense that $\kappa>h_{\text{MLE}}$ , then the MLE does not exist with probability one. Conversely, if $\kappa<h_{\text{MLE}}$ , the MLE asymptotically exists with probability one.
中文速览
逻辑回归（logistic regression）在高维数据中能否算出最大似然估计（MLE，maximum likelihood estimate）是统计推断的核心问题，然而当特征维度 $p$ 与样本量 $n$ 之比较大时，MLE 往往会"跑到无穷远"而不存在，但究竟在什么条件下会发生这一现象，此前缺乏精确的理论刻画。本文证明了：当协变量服从高斯分布、且 $p/n$ 趋向某个极限 $\kappa$ 时，MLE 的存在性经历一个尖锐的"相变"（phase transition）——存在一条由截距项 $\beta_0$ 和信号强度 $\gamma_0$ 共同决定的显式边界曲线 $h_\text{MLE}$，当 $\kappa$ 低于该曲线时 MLE 以概率 1 存在，高于该曲线时以概率 1 不存在，且过渡区间的宽度仅为 $O(n^{-1/2})$。这一边界公式通过凸几何工具严格推导，并在大规模数值模拟中得到精准验证，将 Cover 关于随机标签的经典结论（相变点恰为 $\kappa=1/2$）推广到标签真正依赖特征的一般情形。这一结果为使用逻辑回归的实践者提供了可计算的先验判据，明确指出在何种维度与信号强度组合下似然推断方法从根本上就无法适用。
原文 arXiv:1804.09753;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.09753v1