arXiv：cs.LG（机器学习，全量分类）· Wei Jiang, Dingzhi Yu, Sifan Yang, Wenhao Yang, Zechao Li, Lijun Zhang·· 15 小时前AI 评分33
基于符号的动量方法获得更优收敛保证
Better Convergence Guarantees for Sign-Based Momentum Methods
AI 导读
该论文改进了带动量更新的 signSGD 收敛性分析，证明其在恒定 batch size 且无需额外假设下即可达到 O(T^{-1/4}) 收敛率。在 l2-smoothness 条件下，新结果比此前工作提升 O(d^{1/2})，分布式场景下收敛率也分别优化至 O(d^{1/2}T^{-1/2} + dn^{-1/2}) 和 O(d^{1/4}T^{-1/4})。数值实验验证了方法的有效性。
正文
Abstract:This paper presents an improved analysis for sign-based methods with momentum updates. Traditional sign-based methods obtain a convergence rate of $\mathcal{O}(T^{-1/4})$ under the separable smoothness assumption, but they typically require large batch sizes or assume unimodal symmetric stochastic noise. To address these limitations, we demonstrate that signSGD with momentum can achieve the same convergence rate using constant batch sizes without additional assumptions. We also establish a convergence rate under the $l_2$-smoothness condition, improving upon the result of prior work by a factor of $\mathcal{O}(d^{1/2})$, where $d$ is the problem dimension. Furthermore, we explore sign-based methods in distributed settings and show that the proposed methods yield convergence rates of $\mathcal{O}\left( d^{1/2}T^{-1/2} + dn^{-1/2} \right)$ and $\mathcal{O}\left(d^{1/4}T^{-1/4}\right)$, which outperform the previous results of $\mathcal{O}\left( dT^{-1/4} + dn^{-1/2} \right)$ and $\mathcal{O}\left( d^{3/8}T^{-1/8} \right)$, respectively. Numerical experiments also validate the effectiveness of the proposed methods.
| Subjects: | Optimization and Control (math.OC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2507.12091 [math.OC] |
| (or arXiv:2507.12091v2 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2507.12091 arXiv-issued DOI via DataCite |
Submission history
From: Wei Jiang [view email]
[v1]
Wed, 16 Jul 2025 09:54:08 UTC (383 KB)
[v2]
Thu, 1 Oct 2026 12:53:54 UTC (645 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org