arXiv：cs.LG· Alberto Fern\'andez-Hern\'andez, Cristian P\'erez-Corral, Jose I. Mestre, Manuel F. Dolz, Enrique S. Quintana-Ort\'i·· 4 小时前AI 评分35
为 Adam 选择平衡共享记忆参数 β 的早期记忆选择方法
Early Memory Selection for Balanced Adam
AI 导读
研究提出从短程试运行中为 Adam 选择共享记忆参数 β₁=β₂=β 的方法，选定后在后续完整训练中保持固定。该方法基于 Adam 归一化方向的局部模型，在采样波动与梯度平均延迟之间取得平衡，得出三次记忆规则，其两个系数由少量试运行检查点的梯度探测估计，并联合使用分子与分母以保留其协方差。
正文
Abstract:We propose a method for choosing the shared memory parameter $\beta_1=\beta_2=\beta$ in Adam from a short pilot training. The selected $\beta$ remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared $\beta=0.95$. The mean gap is also 32.3% lower than that of the best constant $\beta$ chosen across all eleven workloads.
| Comments: | Includes theoretical proofs and reproducibility appendices. Code and data: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.08624 [cs.LG] |
| (or arXiv:2610.08624v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08624 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alberto Fernández-Hernández [view email]
[v1]
Tue, 6 Oct 2026 16:22:36 UTC (44 KB)
来源：arXiv：cs.LG · arxiv.org