arXiv：cs.LG· Yiming Lin, Yuxuan Wang·· 7 小时前AI 评分37
Grokking 中正则化驱动动力学的涌现机制研究
Mapping the Emergence of Regularization-Driven Dynamics in Grokking
AI 导读
研究通过在泛化前平台期施加短时固定时长权重衰减扰动，在三个 grokking 任务中发现：扰动对后续泛化时间的影响在平台期早期无序，后期则形成稳定的剂量排序——更强权重衰减使泛化更早、更弱则更晚。扰动与基线检查点间的测试损失壁垒趋于零，但时序效应持续存在，ℓ1 正则化下也观察到类似响应重组。研究者将其称为 grokking 解选择的渠道化。
正文
Abstract:For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalization, providing a window for studying how this selection develops during training. We sweep short, fixed-duration weight decay (WD) perturbations across the pre-generalization plateau and measure how they shift later generalization time. Across three grokking tasks, these shifts are unordered early in the plateau but later form a stable dose ordering before visible generalization, with stronger WD increases leading to earlier generalization and stronger WD decreases leading to later generalization. Test-loss barriers between perturbed and baseline generalization checkpoints collapse toward zero while the ordered timing effects persist. A similar response reorganization is observed under $\ell_1$ regularization in the grokking setting of Junior et al. (2025). Drawing on Waddington's developmental landscape as an analogy, we call this combination of increasingly constrained solution selection and persistent dose-ordered timing shifts the canalization of grokking solution selection. Together, our response maps and loss-barrier measurements reveal a dynamical reorganization before visible generalization that is consistent with the theoretical picture of regularization-driven motion along a stable slow manifold (Boursier et al., 2025).
| Comments: | 23 pages, 13 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.25813 [cs.LG] |
| (or arXiv:2608.25813v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.25813 arXiv-issued DOI via DataCite |
Submission history
From: Yiming Lin [view email]
[v1]
Wed, 26 Aug 2026 13:58:14 UTC (2,260 KB)
[v2]
Fri, 4 Sep 2026 17:07:12 UTC (2,421 KB)
[v3]
Tue, 6 Oct 2026 01:21:44 UTC (2,711 KB)
来源：arXiv：cs.LG · arxiv.org