arXiv：cs.LG· Vihaan Paka-Hegde·· 2 天前AI 评分33
Adam 距离自然梯度下降有多远？
How Far is Adam from Natural Gradient Descent?
AI 导读
一项 arXiv 研究把 Adam 的完整更新规则（含动量）视为带对角截断、经验标签替换和时间滞后的对角经验 Fisher 近似，用尺度不变的 γ(Δθ) 度量其在四类损失景观上与真实 NGD 的几何偏差。
正文
Abstract:Adam is the standard optimizer in deep learning, yet its geometric relationship to natural gradient descent (NGD) contains unresolved questions. We study Adam's full update rule, including momentum, as a diagonal empirical Fisher approximation subject to diagonal truncation, empirical label substitution, and temporal lag. Using the scale-invariant $\gamma(\Delta\theta)$ metric, we measure Adam's geometric deviation from true NGD across four loss landscapes: well-conditioned linear regression, ill-conditioned linear regression, logistic regression, and a non-convex small neural network. Adam's geometric trajectory is context-dependent. Deviation remains low in well-conditioned settings but rises significantly under ill-conditioning, reaching misalignments of $\approx 10^3$ in the neural network. Higher geometric drift correlates with slower initial optimization but does not degrade final objective minimization; Adam consistently reaches low loss. Furthermore, the improved empirical Fisher (iEF) tracks more stable paths than the standard empirical Fisher (EF), which frequently oscillates or diverges. Our results suggest Adam's practical optimization power may stem from a balance of structural approximation errors and momentum smoothing rather than close tracking of the natural gradient path.
| Comments: | 9 pages, 4 figures, 2 tables |
| Subjects: | Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE) |
| MSC classes: | 68T07 |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2610.00004 [cs.LG] |
| (or arXiv:2610.00004v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00004 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.5281/zenodo.20466393
DOI(s) linking to related resources |
Submission history
From: Vihaan Paka-Hegde [view email]
[v1]
Sat, 30 May 2026 21:59:23 UTC (169 KB)
来源：arXiv：cs.LG · arxiv.org