arXiv：cs.LG· Yiman Fong, Heng Yang·· 2 天前AI 评分44
Adam 在稳定性边缘：自适应反馈、可证明振荡与梯度反转
Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal
AI 导读
研究揭示全批量 Adam 的稳定性边缘（EoS）现象源于其二阶矩自适应构成的负反馈机制，该机制通过"活跃曲率"将动力学推向稳定边界。该机制预测 Adam 在边缘附近出现梯度反转，即连续梯度反复指向近乎相反方向，并在全连接网络、ResNet、ViT、LSTM、GPT-2 medium 及 Adam 系列优化器上得到验证。对迭代取平均可抑制这种快速振荡，得到更平滑、更低的损失曲线。
正文
Abstract:The edge-of-stability (EoS) phenomenon of full-batch Adam has been widely observed, yet its underlying dynamical mechanism remains poorly understood. In this paper, we identify Adam's second-moment adaptation as a negative-feedback mechanism that drives the dynamics toward the stability boundary. We characterize this mechanism through the *active curvature*, namely, the preconditioned curvature along the preconditioned gradient direction, and establish rigorous characterizations in progressively richer settings: rank-one quadratics with momentum, diagonal quadratics, on which the active curvature separates from the sharpness, and general objectives. Importantly, the mechanism predicts *gradient reversal* of full-batch Adam near the edge: consecutive gradients repeatedly point in nearly opposite directions, as we observe across fully connected networks, ResNets, ViTs, LSTMs, GPT-2 medium, and Adam-family optimizers. Consistent with this picture, averaging iterates suppresses these fast oscillations and produces smoother and lower loss curves. Together, these results provide an important first step towards fully understanding the dynamical behavior of Adam's EoS through active curvature and gradient reversal.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC) |
| Cite as: | arXiv:2608.20638 [cs.LG] |
| (or arXiv:2608.20638v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.20638 arXiv-issued DOI via DataCite |
Submission history
From: Yiman Fong [view email]
[v1]
Fri, 21 Aug 2026 00:22:11 UTC (78 KB)
[v2]
Thu, 1 Oct 2026 10:25:00 UTC (3,929 KB)
来源：arXiv：cs.LG · arxiv.org