arXiv：cs.CL· Nathan Godey, Yoav Artzi·· 6 小时前
Copy-Paste Language Models：为 LLM 增加 Copy-Paste 头提升复制能力
Copy-Paste Language Models
AI 导读
研究者提出 Copy-Paste Language Models（CPLMs），在传统语言建模（LM）头之外增加一个轻量级 Copy-Paste 头，提升 LLM 的上下文复制能力。团队预训练了 135M 到 1.7B 参数的模型，在所有规模和骨干网络上均取得提升，训练与推理开销可忽略，并在长上下文和检索场景中提供新的工作点。CPLMs 通过替代梯度路径部分缓解了 LM 头的梯度瓶颈。
正文
Abstract:Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19247 [cs.CL] |
| (or arXiv:2610.19247v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nathan Godey [view email]
[v1]
Tue, 13 Oct 2026 22:03:18 UTC (750 KB)
来源：arXiv：cs.CL · arxiv.org