arXiv：cs.LG· Stephen Y Zhang, Gabriel Peyr\'e·· 4 小时前AI 评分35
两层线性网络训练的全局指数收敛性
Global Exponential Convergence of Two-Layer Linear Network Training
AI 导读
研究证明宽两层线性网络在平滑 Polyak-Lojasiewicz 预测损失下可实现显式速率的全局指数收敛，梯度流在因子上精确闭合为神经元律协方差的有限维 Bures 流。当初始协方差满足谱支撑间隙条件时，损失以至少 4σ²κ 的线性速率收敛至全局最小值，κ 为 PL 常数；该速率在有限宽度采样下保持稳定，并扩展至深度线性 ResNet 及 heavy-ball 动量情形。
正文
Abstract:We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losses. Gradient flow in the factors closes exactly in terms of a finite-dimensional Bures flow of the neuron law covariance, in which the predictor dynamics are preconditioned by hidden covariance blocks. Mean-field conservation laws provide uniform spectral lower bounds on the hidden preconditioning blocks when the initial covariance satisfies a spectral support gap condition. This condition encompasses positive definiteness while still allowing for singular initializations. For an initial covariance $\Sigma_0 = \sigma^2 \mathrm{Id}$, the loss converges to the global minimum with linear rate at least $4\sigma^2\kappa$, where $\kappa$ is the PL constant. We establish stability of this rate under finite-width sampling, as well as global convergence of factor gradient descent for an explicit stepsize interval depending on smoothness, the initial loss, and conserved spectral margins. Our argument extends layerwise to deep linear ResNets, subject to a residual-path bound. In the case of heavy-ball momentum, training dynamics close instead over positions and velocities in terms of a lifted phase covariance. Linear convergence holds under an explicit condition on the energy and damping, specifying a window of admissible dampings. For two-scale white initializations, this interval is nonempty for sufficiently large position scales, with a fixed initial loss gap and velocity covariance. Numerical experiments illustrate the covariance geometry and compare the predicted and observed rates.
| Subjects: | Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.09356 [cs.LG] |
| (or arXiv:2610.09356v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09356 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Stephen Zhang [view email]
[v1]
Wed, 7 Oct 2026 03:13:18 UTC (560 KB)
来源：arXiv：cs.LG · arxiv.org