arXiv：cs.LG· Yaroslav Gusev, Eugene Golikov, Dmitry Yarotsky·· 4 小时前
小初始化下的「精度井」：分类精度非单调动力学研究
The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
AI 导读
研究揭示小初始化下分类训练存在「精度井」现象：精度早期快速上升后下降，很久之后才恢复，该现象在深度线性网络、MNIST 上的全连接 ReLU 网络、CIFAR-10 上的深度卷积网络及合成任务中均被观察到。
正文
Abstract:We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.
| Comments: | 124 pages, 11 figures. Submitted to ICLR-2027: this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20439 [cs.LG] |
| (or arXiv:2610.20439v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20439 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaroslav Gusev [view email]
[v1]
Wed, 14 Oct 2026 16:28:22 UTC (1,553 KB)
来源：arXiv：cs.LG · arxiv.org