arXiv：cs.LG· El Mahdi Chayti, Mohamad Fakhouri·· 13 小时前
RanSOM：用随机化步长实现免 Lipschitz Hessian 的二阶动量优化
RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
AI 导读
RanSOM 提出通过将步长随机化来消除二阶动量校正中对 Lipschitz Hessian 或额外前向反向传播的依赖。当步长服从均值为 η 的指数分布时，一步 HVP 即可无偏估计梯度变化，且该 HVP 与下一次梯度共享前向传播。
正文
Abstract:Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extra forward and backward passes at an auxiliary point. We show that randomizing the step length removes this trade-off: if the step is exponentially distributed with mean $\eta$, a Stein-type identity makes $\eta$ times one HVP at the new iterate an estimate of the gradient change that is unbiased in expectation over the step, and this HVP shares its forward pass with the next gradient. The resulting method, RanSOM-E, works with normalized, sign, and Muon-type steps, and a Beta-distributed variant, RanSOM-B, keeps Frank-Wolfe iterates feasible. For objectives with Lipschitz gradient and heavy-tailed gradient and Hessian noise with tail indices $p, q \in (1,2]$, RanSOM-E attains the optimal oracle-complexity exponents for every pair $(p,q)$, including $p \neq q$, without a Lipschitz Hessian or clipping ($\mathcal{O}(\varepsilon^{-3})$ for $p = q = 2$). On CIFAR-10 with ResNet-20, we detect no accuracy difference from existing second-order momentum methods, and its mean update time is lower than that of the auxiliary-point method.
| Subjects: | Optimization and Control (math.OC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.06824 [math.OC] |
| (or arXiv:2602.06824v3 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2602.06824 arXiv-issued DOI via DataCite |
Submission history
From: El Mahdi Chayti [view email]
[v1]
Fri, 6 Feb 2026 16:09:36 UTC (93 KB)
[v2]
Thu, 14 May 2026 19:41:27 UTC (99 KB)
[v3]
Fri, 9 Oct 2026 17:30:50 UTC (354 KB)
来源：arXiv：cs.LG · arxiv.org