arXiv：cs.LG· Matias D. Cattaneo, Boris Shigida·· 4 小时前AI 评分36
带修改损失的动量梯度下降：细粒度分析
Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis
AI 导读
研究证明，Polyak 重球动量（HB）在步长 h 足够小时，于指数吸引不变流形上等价于带修改损失的普通梯度下降。作者将该修改损失描述到 O(h^R) 误差（任意有限阶 R），并给出全局轨迹近似界 O(h^R)，覆盖全批量与 mini-batch HB。
正文
Abstract:We analyze gradient descent with Polyak (1964) heavy-ball momentum (HB) whose fixed momentum hyperparameter $\beta \in (0, 1)$ provides exponential decay of memory. Building on Kovachki and Stuart (2021), we prove that on an exponentially attractive invariant manifold the algorithm is exactly plain gradient descent with a modified loss, provided that the step size $h$ is small enough. Although the modified loss does not admit a closed-form expression, we describe it up to $O(h^{\mathcal{R}})$-errors for arbitrary finite order $\mathcal{R}$, and prove global (finite "time" horizon) trajectory approximation bounds $O(h^{\mathcal{R}})$. We then conduct a fine-grained analysis of the combinatorics underlying the memoryless approximations of HB, in particular, finding a rich family of polynomials in $\beta$ hidden inside which include and lie coefficient-wise in between Eulerian and Narayana polynomials. We prove that these polynomials are $h$-polynomials of certain graph-associahedra. As corollaries of the main results, we derive continuous modified equations of arbitrary finite approximation order (with rigorous bounds) and the principal flow that approximates the HB dynamics, generalizing Rosca et al. (2023). Approximation theorems cover both full-batch and mini-batch HB. The results shed new light on the main features of HB and outline a roadmap for similar analysis of other optimization algorithms.
| Subjects: | Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC); Computation (stat.CO); Machine Learning (stat.ML) |
| Cite as: | arXiv:2509.08483 [cs.LG] |
| (or arXiv:2509.08483v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.08483 arXiv-issued DOI via DataCite |
Submission history
From: Boris Shigida [view email]
[v1]
Wed, 10 Sep 2025 10:47:54 UTC (95 KB)
[v2]
Wed, 7 Oct 2026 04:01:52 UTC (113 KB)
来源：arXiv：cs.LG · arxiv.org