arXiv：cs.LG· Forough Shirin Abkenar, Haoshuo Fu, Liang Feng, Ning Yan, Masood S. Mortazavi·· 3 小时前
研究追踪 LLM 语言习得中的形态、语义与分歧：基于 Pythia 和 OLMo 中间检查点
Tracing Morphology, Semantics, and Divergence in Language Acquisition
AI 导读
研究利用 Pythia 和 OLMo 等开源模型的中间检查点，在 teacher-forcing 评估下追踪 LLM 训练过程中的语言习得轨迹。结果显示，模型习得形态结构的时间远早于语义嵌入趋于稳定的时间；训练数据集规模对分歧识别有正向影响，而预训练中的共现和推理中的重复则对分歧识别与关联链接生成有负向影响。研究聚焦形态结构习得、嵌入漂移、重复主导和共现效应四项核心特征。
正文
Abstract:Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19041 [cs.LG] |
| (or arXiv:2610.19041v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19041 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Forough Shirin Abkenar [view email]
[v1]
Tue, 13 Oct 2026 18:29:52 UTC (21,804 KB)
来源：arXiv：cs.LG · arxiv.org