arXiv：cs.LG· Yiding Song, Hanming Ye·· 13 小时前
模型容量如何决定 Grokking：记忆与泛化速度的竞争
Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
AI 导读
一项信息论研究揭示，Grokking 并非模型大到能记住训练集就立即出现，而是记忆速度 T_mem(P) 与泛化速度 T_gen(P) 两种可测量时间尺度竞争的结果，两者均为参数量 P 的函数。研究在模算术任务上估算这两个时间尺度，发现 Grokking 出现在两者相交的参数量级附近。该框架还提出根据模型容量与数据集复杂度预测记忆速度的经验模型，印证了更大模型记忆更快的观察。
正文
Abstract:Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions. However, despite a known dependence between grokking and model size, how model capacity shapes grokking remains an open question. We give an information-theoretic account of this relationship on the task of modular arithmetic, showing that grokking does not immediately occur when a model becomes large enough to memorise the training set, but rather emerges as the outcome of a competition between two measurable timescales: a memorisation speed $T_{\text{mem}}(P)$ and a generalisation speed $T_{\text{gen}}(P)$, both of which are functions of model parameter count $P$. Adapting the information capacity framework of Morris et al. (2025), we estimate $T_{\text{mem}}(P)$ on random-label data of equivalent complexity and $T_{\text{gen}}(P)$ on the modular task itself, and show that grokking emerges close to the parameter scale where these timescales intersect. The framework also suggests an empirical model for predicting memorisation speed given model capacity and dataset complexity, recovering the previously reported empirical observation that larger models memorise faster. Overall, we motivate the formalisation of different learning timescales as important abstractions to study when explaining how model capacity shapes grokking on algorithmic tasks.
| Comments: | Updated references section |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.09724 [cs.LG] |
| (or arXiv:2605.09724v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09724 arXiv-issued DOI via DataCite |
Submission history
From: Yiding Song [view email]
[v1]
Sun, 10 May 2026 19:47:39 UTC (477 KB)
[v2]
Fri, 9 Oct 2026 09:06:32 UTC (477 KB)
来源：arXiv：cs.LG · arxiv.org