arXiv：cs.LG· Yongding Tian, Zaid Al-Ars, Maksim Kitsak, Peter Hofstee·· 2 天前AI 评分41
低损失区域错位导致 Grokking：一项基于模式连通性的分析
Misalignment of Low-Loss Regions Causes Grokking
AI 导读
研究提出基于模式连通性与低损失区域几何的分析框架，发现标准模运算设置并非总产生 Grokking：在保持对称性的训练/验证划分下出现验证性能不恢复的稳定 anti-grokking 反例。该框架进一步表明，Grokking 源于训练与验证划分诱导的低损失区域错位；一旦区域对齐，仅靠训练超参数无法产生 Grokking，动力学坍缩为可训练或不可训练行为。
正文
Abstract:Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.
| Comments: | 23 pages, 23 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.00620 [cs.LG] |
| (or arXiv:2610.00620v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00620 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yongding Tian [view email]
[v1]
Wed, 30 Sep 2026 19:22:20 UTC (2,371 KB)
来源：arXiv：cs.LG · arxiv.org