arXiv：cs.LG· Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu·· 4 小时前
Muon 为何失去谱隐式偏置：从谱范数到 Frobenius 的转变
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
AI 导读
研究证明 Muon 在实现中使用有限 Newton-Schulz 迭代和正稳定常数近似正交化时，在线性可分数据的分类任务中会选择 Frobenius 而非谱 max-margin 方向。
正文
Abstract:Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20112 [cs.LG] |
| (or arXiv:2610.20112v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20112 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yongtao Wu [view email]
[v1]
Wed, 14 Oct 2026 12:42:35 UTC (80 KB)
来源：arXiv：cs.LG · arxiv.org