Jamming in multilayer supervised learning models
Silvio Franz LPTMS, Université Paris-Sud 11, UMR 8626 CNRS, Bât. 100, 91405 Orsay Cedex, France Sungmin Hwang LPTMS, Université Paris-Sud 11, UMR 8626 CNRS, Bât. 100, 91405 Orsay Cedex, France Pierfrancesco Urbani Institut de physique théorique, Université Paris Saclay, CNRS, CEA, F-91191 Gif-sur-Yvette
Abstract
Critical jamming transitions are characterized by an astonishing degree of universality. Analytic and numerical evidence points to the existence of a large universality class that encompasses finite and infinite dimensional spheres and continuous constraint satisfaction problems (CCSP) such as the non-convex perceptron and related models. In this paper we investigate multilayer neural networks (MLNN) learning random associations as models for CCSP which could potentially define different jamming universality classes. As opposed to simple perceptrons and infinite dimensional spheres, which are described by a single effective field in terms of which the constraints appear to be one-dimensional, the description of MLNN, involves multiple fields, and the constraints acquire a multidimensional character. We first study the models numerically and show that similarly to the perceptron, whenever jamming is isostatic, the sphere universality class is recovered, we then write the exact mean-field equations for the models and identify a dimensional reduction mechanism that leads to a scaling regime identical to the one of infinite dimensional spheres. We suggest that this mechanism could be g
中文速览
多层神经网络学习随机样本时，在约束饱和的"堵塞转变"（jamming transition）处究竟属于哪个普适类，是否会因为隐藏层数量增多而产生新的普适类，这是本文要回答的核心问题。研究者以奇偶机、委员会机和ReLU两层网络为代表模型，既通过数值模拟测量了堵塞点附近的力分布与间隙分布幂律指数，又推导了精确的平均场（replica）方程，分析其在堵塞点附近的标度行为。结果表明，尽管多层网络的鞍点方程天然是多维偏微分方程，但在临界标度区间存在一种"维度约化"机制，使方程退化为与无限维硬球体系完全相同的一维形式，数值上测得的临界指数（θ≈0.42，γ≈0.41）也与无限维硬球的精确值吻合。这一发现意味着，只要堵塞点同时满足等静定（isostaticity）和热力学边际稳定两个条件，无论网络结构多复杂，其堵塞普适类都归属于硬球填充这一单一普适类，并为理解有限维堵塞现象中观察到的普适性提供了新的理论视角。
原文 arXiv:1809.09945;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.09945v3