arXiv：cs.LG· Ali Janati, Kaoutar El Maghraoui, Anass Belfatmi·· 4 小时前AI 评分35
Muon 训练的 Transformer 在表征-读出接口处的 Post-Grokking 崩塌
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
AI 导读
Muon 训练的模算术 Transformer 会在保留线性可解码任务信息的同时丢失准确率，相邻交换将五个未归一化失败定位到 AdamW 读出更新。将实际读出位移乘以较大的特征均值会产生跨输入共享的类别相关 logit 偏移，几乎复现每个失败。仅用训练集训练的解码器可恢复 98.20-100% 的留出准确率；修正交叉熵导数误差可使五个匹配分支稳定至第 100,000 步。
正文
Abstract:Muon-trained modular-arithmetic transformers can lose accuracy while retaining linearly decodable task information. Adjacent swaps localize five captured unnormalized failures to AdamW readout updates. Multiplying the actual readout displacement by the large feature mean produces a class-dependent logit offset shared across inputs that nearly reproduces each failure. Training-only decoders recover 98.20-100% held-out accuracy. Correcting cross-entropy derivative errors stabilizes five matched branches through step 100,000; four prospective accurate-CE RMS runs fail through embedding updates.
| Comments: | 38 pages, 11 figures. Revised manuscript with expanded experiments and analysis. Preprint. Under review |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.07436 [cs.AI] |
| (or arXiv:2608.07436v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.07436 arXiv-issued DOI via DataCite |
Submission history
From: Ali Janati [view email]
[v1]
Fri, 7 Aug 2026 17:21:49 UTC (199 KB)
[v2]
Tue, 6 Oct 2026 06:08:27 UTC (821 KB)
来源：arXiv：cs.LG · arxiv.org