LionMuon：交替谱下降与符号下降的高效训练优化器

This paper has been withdrawn by Arman Bolatov

No PDF available, click to view other formats

Abstract:Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every $P$ iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per $P$ steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with $P=2$ and $P=5$ reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: this https URL

Comments:	I mistakenly created a new submission instead of editing the existing one
Subjects:	Machine Learning (cs.LG)
Cite as:	arXiv:2609.35297 [cs.LG]
	(or arXiv:2609.35297v3 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2609.35297 arXiv-issued DOI via DataCite

Submission history

From: Arman Bolatov [view email]
[v1] Mon, 28 Sep 2026 14:39:14 UTC (709 KB)
[v2] Mon, 5 Oct 2026 14:36:39 UTC (710 KB)
[v3] Tue, 6 Oct 2026 08:41:17 UTC (1 KB) (withdrawn)