arXiv：cs.LG· Mohnish Harwani·· 3 小时前
平坦极小值能带来更好泛化吗？Grokking 中的算法分离
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
AI 导读
研究用 SAM 偏置训练并不能可靠诱发 Grokking 中从记忆到泛化的转变，即便产生了更平坦的解。但 SAM 与 weight decay 结合时，可将泛化转变在 epoch 级别加速最多 4 倍。作者用两层 ReLU 模型从理论上证明，平坦性本身无法区分记忆解与泛化解，而 weight decay 偏好泛化解。
正文
Abstract:Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11206 [cs.LG] |
| (or arXiv:2610.11206v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11206 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mohnish Harwani [view email]
[v1]
Thu, 8 Oct 2026 04:06:50 UTC (1,582 KB)
来源：arXiv：cs.LG · arxiv.org