arXiv：cs.LG（机器学习，全量分类）· Joohwan Ko, Tetiana Parshakova, Diana Cai, Robert M. Gower·· 14 小时前AI 评分40
SoftServe：面向深度学习的可扩展拟牛顿方法
SoftServe: A Scalable Quasi-Newton Method for Deep Learning
AI 导读
研究者提出 SoftServe，一族无需线搜索或临时曲率修正的拟牛顿（QN）方法，可从变分目标导出正定曲率估计，即使存在负曲率也成立。其对角与 Kronecker 分解变体按构造保持正定性，可扩展至大规模神经网络，并用稳定的耦合 Newton-Schulz 迭代以 GPU 友好的矩阵乘法替代昂贵的矩阵分解。
正文
Abstract:Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.02182 [cs.LG] |
| (or arXiv:2610.02182v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02182 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joohwan Ko [view email]
[v1]
Thu, 1 Oct 2026 17:58:24 UTC (1,803 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org