arXiv：cs.LG· Sourabh Kasliwal·· 4 小时前AI 评分37
小 Transformer 算术推理的算法草稿本与课程分级研究
Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
AI 导读
一项 arXiv 研究用约 1060 万非嵌入参数（总计 4930 万）的 Tiny Transformer 在合成数据上训练四则运算草稿本，发现连续序列打包可修复 dataloader 填充导致的 83% 梯度饥饿问题（准确率从 40% 跌至 1%），语言预训练是必要前提（缺失时准确率 ≤2.0%）。
正文
Abstract:Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division. In this paper, we investigate the mechanics of multi-step arithmetic in compact "Tiny" Transformers (~10.6M non-embedding parameters, 49.3M total) trained on synthetic data across four basic operations (+, -, *, /) unrolled as step-by-step scratchpads. First, we establish the necessary training foundations: (1) dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remediated via continuous sequence packing; (2) linguistic pretraining is an essential prerequisite (<= 2.0% without it); and (3) modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) substantially improve additive reasoning over baseline GPT-2. Second, we demonstrate that algorithmic scratchpad formulation directly dictates success. Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division from 4.0% to 86.7% accuracy on a 4,000-problem held-out benchmark. In contrast, multi-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single-digit sub-products and place-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step without pairwise intermediate accumulation. Finally, we identify two key boundaries: performance collapses to 0.00% on unseen 4-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%.
| Comments: | 16 pages, 5 figures. Code and benchmark datasets available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09003 [cs.LG] |
| (or arXiv:2610.09003v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09003 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sourabh Kasliwal [view email]
[v1]
Tue, 6 Oct 2026 18:58:13 UTC (118 KB)
来源：arXiv：cs.LG · arxiv.org