arXiv：cs.LG· Priya Kasimbeg (Alice), Ahmed Khaled (Alice), Aahlad Chandrabhatta (Alice), Yifan Wang (Alice), Andy Liu (Alice), Niccol\`o Ajroldi (Alice), David Tweedle (Alice), Xingyu (Alice), Yang, Isaac Patole, Frank Schneider, George E. Dahl·· 13 小时前
AlgoPerf 基准更新：Schedule-Free AdamW 在自调参规则下排名第一
Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf
AI 导读
更新版 AlgoPerf V1 基准在固定硬件的九个负载上评测新训练算法的实际耗时达标表现，其中自调参规则要求提交方案在所有负载上使用单一配置。在 PyTorch 和 JAX 的 13 份提交中，Schedule-Free AdamW 排名第一。报告还分析了步效率与运行时执行的权衡，超参数敏感性与保留分析显示学习率和动量在不同领域和算法间高度敏感。
正文
Authors:Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta, Yifan Wang, Andy Liu, Niccolò Ajroldi, David Tweedle, Xingyu (Alice)Yang, Isaac Patole, Frank Schneider, George E. Dahl
Abstract:Evaluating deep learning optimizer is challenging because confounding factors such as hyperparameter tuning and optimizer-model co-design often obscure true algorithmic performance. We present results of the updated AlgoPerf V1 benchmark, in which we evaluate a new set of training algorithms on wall-clock time-to-target across nine workloads on fixed hardware. This report focuses on the results of the benchmark's self-tuning ruleset, which requires submissions to use a single configuration across all workloads. Out of the 13 submissions in PyTorch and JAX, Schedule-Free AdamW ranked first. In addition to the wall-clock scores, we also analyze the trade-offs between step efficiency and wall-clock execution, revealing that matrix-preconditioned and two-loop methods achieve good sample efficiency but require more aggressive computational optimizations to be competitive in runtime. Finally, our hyperparameter saliency and retention analyses show that the learning rate and momentum are highly sensitive across domains and algorithms, highlighting the need for methods that are more robust against hyperparameter tuning and can generalize well to new tasks.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13036 [cs.LG] |
| (or arXiv:2610.13036v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13036 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Priya Kasimbeg [view email]
[v1]
Fri, 9 Oct 2026 05:47:51 UTC (1,187 KB)
来源：arXiv：cs.LG · arxiv.org