arXiv：cs.CL· Fanfei Li, Jana Zeller, Manuel Prada-Corral, Thadd\"aus Wiedemer, Prasanna Mayilvahanan, Ryan Cotterell, Wieland Brendel·· 3 小时前AI 评分42
LittleLearner：在教学法控制的知识暴露下训练的语言模型
LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
AI 导读
研究者构建 LittleCurriculum——一个 88B token、仅覆盖美国小学五年级以下内容的预训练语料，并据此从零训练出 5B 参数的 LittleLearner，其知识与能力边界对应可解释的课程大纲。通过后训练和上下文学习注入新知识，只能让模型更好利用已有知识，无法提升超出范围的能力。
正文
Abstract:Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LittleCurriculum, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LittleCurriculum yields LittleLearner, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LittleCurriculum and LittleLearner as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LittleLearner better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.13545 [cs.CL] |
| (or arXiv:2608.13545v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.13545 arXiv-issued DOI via DataCite |
Submission history
From: Jana Zeller [view email]
[v1]
Thu, 13 Aug 2026 17:56:12 UTC (2,635 KB)
[v2]
Wed, 7 Oct 2026 10:06:13 UTC (2,635 KB)
来源：arXiv：cs.CL · arxiv.org