arXiv：cs.LG· Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky·· 5 小时前AI 评分43
语言模型预训练中的 Grokking 类比：追踪延迟的语法泛化
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
AI 导读
研究者提出基于曝光的框架，在 LLM 预训练中研究类 Grokking 动态，并以 BLiMP 最小对为评测基础，按关键短语是否出现在预训练窗口划分 proxy-train 与 proxy-validation。在五种语法现象上均观察到延迟泛化，泛化后语法概念向量对语法可接受性的预测性更强、占据更高维子空间，且关键 token 对相关上下文 token 的注意力集中在少数注意力头上。
正文
Abstract:Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.
| Comments: | 18 pages, 10 figures, 9 tables; Accepted to AACL-IJCNLP 2026 Main Conference |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.00230 [cs.LG] |
| (or arXiv:2606.00230v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.00230 arXiv-issued DOI via DataCite |
Submission history
From: Sherin Muckatira [view email]
[v1]
Fri, 29 May 2026 18:04:52 UTC (2,741 KB)
[v2]
Fri, 2 Oct 2026 16:06:27 UTC (3,627 KB)
来源：arXiv：cs.LG · arxiv.org