arXiv：cs.LG· Hinata Harada, Hideaki Iiduka·· 7 小时前AI 评分26
使用递增批量大小与衰减学习率的 Sharpness-Aware Minimization 算法的收敛性
Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
AI 导读
该论文从理论上证明了 GSAM 算法在采用递增批量大小或衰减学习率（如余弦退火、线性学习率）时的收敛性。数值实验对比显示，使用递增批量大小相比恒定批量大小和学习率，能达到更低的 worst-case ℓ∞ 自适应锐度。SAM 及其变体 GSAM 通过寻找平坦局部极小值提升深度神经网络泛化能力。
正文
Abstract:The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural network models by finding flat local minima of the empirical loss in training. Meanwhile, it has been shown theoretically and practically that increasing the batch size or decaying the learning rate avoids sharp local minima of the empirical loss. In this paper, we consider the GSAM algorithm with increasing batch sizes or decaying learning rates, such as cosine annealing or linear learning rate, and theoretically show its convergence. Moreover, we numerically compare SAM (GSAM) with and without an increasing batch size and conclude that using an increasing batch size { achieves a lower worst-case $\ell_\infty$ adaptive sharpness} than compared with using a constant batch size and learning rate.
| Subjects: | Machine Learning (cs.LG); Optimization and Control (math.OC) |
| Cite as: | arXiv:2409.09984 [cs.LG] |
| (or arXiv:2409.09984v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2409.09984 arXiv-issued DOI via DataCite |
Submission history
From: Hinata Harada [view email]
[v1]
Mon, 16 Sep 2024 04:27:11 UTC (810 KB)
[v2]
Tue, 6 Oct 2026 09:03:23 UTC (740 KB)
来源：arXiv：cs.LG · arxiv.org