HuggingFace Daily Papers·· 2 天前
混合架构中的竞争学习现象：分阶段训练让注意力学会检索
Competing Learning in Hybrid Architectures
AI 导读
论文揭示混合架构中的"竞争学习"现象：不同 mixer 争夺同一函数，任一层学会后即压制其他层的学习信号。层归因分析显示现代循环层比 softmax 注意力更快学会检索函数，导致注意力多数情况下学不会检索。据此提出分阶段预训练流程，在训练中途冻结循环层权重更新，最高 1B 参数模型在语言建模损失、常识推理及多种检索任务上均有明显提升。
正文 · 原文
Abstract:Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.26830 [cs.CL] |
| (or arXiv:2610.26830v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26830 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xingyu Shen [view email]
[v1]
Mon, 19 Oct 2026 17:55:55 UTC (302 KB)
来源：HuggingFace Daily Papers · arxiv.org