arXiv：cs.LG· Simin Fan, Alireza Abdollahpoorrostam, Martin Jaggi·· 3 小时前AI 评分39
OptiSelect：优化器如何塑造数据课程？
OptiSelect: How does the Optimizer Shape Data Curriculum?
AI 导读
研究者提出优化器感知的在线数据选择范式 OptiSelect，首次系统研究优化器如何影响数据选择。理论证明 Lion 和 Muon 的符号型与极切向预条件器会遭遇可判别性崩塌，限制选择增益，而 AdamW、Sophia 等对角自适应优化器具有更优上界。在 124M 和 720M 模型上的预训练实验与理论一致，即使以 Muon 为优化器，AdamW 的对角自适应评分几何仍最强。
正文
Abstract:Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient before it updates model parameters. We formalize this optimizer-aware selection paradigm as OptiSelect and present the first systematic study of how the optimizer shapes data selection. Our theory establishes a selection gain principle in which the advantage of online selection is governed by the discriminability of the optimizer-induced utility scores. We prove that sign-based and polar-tangential preconditioners of Lion and Muon would suffer from a discriminability collapse which caps attainable gains from OptiSelect, whereas diagonal-adaptive optimizers such as AdamW and Sophia admit strictly better upper bounds. The proposed principle also yields a quantitative derivation of the optimal candidate oversampling ratio. Pretraining experiments on 124M and 720M models are consistent with our theoretical analysis and show that AdamW's diagonal-adaptive scoring geometry remains the strongest scoring geometry even with Muon as optimizer. We further demonstrate that OptiSelect retains its benefits under data rephrasing, a technique used in modern data processing pipelines. Our findings provide theoretical foundations and practical guidance for co-designing optimizers and data selection in LLM pretraining.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.03432 [cs.LG] |
| (or arXiv:2610.03432v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03432 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Simin Fan [view email]
[v1]
Fri, 2 Oct 2026 15:13:16 UTC (899 KB)
来源：arXiv：cs.LG · arxiv.org