arXiv：cs.LG· Lexiao Lai, Tianyi Lin, Jiayu Zhang·· 13 小时前
正交化动量在非光滑优化中的失效与修复：MAGD 方法
Nonsmooth Optimization via Orthogonalized Momentum
AI 导读
研究证明，对任意固定动量因子 β∈[0,1)，Muon 在凸 Lipschitz 目标上从几乎所有初始化都无法逼近全局最优解，即使迭代有界；障碍在于固定动量而非正交化本身。
正文
Abstract:Modern real application problems involve matrix-valued parameters, yet conventional optimizers treat them as vectors, thereby motivating matrix-aware methods that exploit input-output geometry, such as Muon which orthogonalizes the momentum matrices before parameter updates. Its empirical success raises a conceptual question: can orthogonalized momentum remain effective beyond smooth optimization? This paper studies this question for locally Lipschitz functions using a generalized derivative framework compatible with backpropagation. Our first contribution is to identify a key limitation: for every fixed momentum factor $\beta\in[0,1)$, Muon can fail to approach the global optimal solution of a convex Lipschitz objective from almost every initialization, when step sizes adapt to the full gradient history. The failure can occur even along bounded iterates. Our example is inspired by the one of Parshakova et al. which only covers $\beta\in[0,\frac{1}{2})$. Then, we show that the obstruction lies in fixed momentum rather than orthogonalization. Indeed, when the momentum factor is adaptive and approaches 1 together with a vanishing step size, Muon recovers asymptotic convergence for nonconvex nonsmooth optimization under the boundedness and regularity conditions. Moreover, we propose MAGD, which combines orthogonalized momentum with gradient, weighted based on their relative progress. MAGD retains asymptotic convergence in nonconvex settings and achieves an $O(\min\{m,n\}\epsilon^{-2})$ rate in convex settings. A lower bound shows the optimal dimension dependence. Experiments on synthetic problems, image classification, and LLM pretraining show MAGD is a simple and practical alternative to Muon. Together, our results characterize when orthogonalized momentum fails without smoothness and how it can be made reliable and we hope that the analysis may be useful more broadly.
| Comments: | 32 pages, 3 figures |
| Subjects: | Optimization and Control (math.OC); Machine Learning (cs.LG) |
| MSC classes: | 90C56, 90C25, 90C60 |
| Cite as: | arXiv:2609.13677 [math.OC] |
| (or arXiv:2609.13677v2 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2609.13677 arXiv-issued DOI via DataCite |
Submission history
From: Jiayu Zhang [view email]
[v1]
Sat, 12 Sep 2026 03:08:36 UTC (568 KB)
[v2]
Fri, 9 Oct 2026 00:40:38 UTC (568 KB)
来源：arXiv：cs.LG · arxiv.org