arXiv：cs.LG· Yaomengxi Han, Debarghya Ghoshdastidar·· 7 小时前
Transformer 在无界输入上的训练动态：注意力学习可能滞后于输出层学习
Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
AI 导读
针对浅层 Transformer 从无界长度输入中识别 star-free 语言的任务，研究者证明三阶段梯度下降才能学会该语言：先更新输出层，再更新注意力，最后微调输出层。仅训练注意力层或输出层、或先单步更新注意力再微调输出层，均无法学会。实验显示"注意力学习滞后于输出层学习"在语言识别、回归和上下文学习任务中同样有益。
正文
Abstract:The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and ability of the attention mechanism to focus on relevant input features. However, there is limited understanding of the training dynamics of shallow transformers, specifically how to decouple the learning of the attention and the output (feedforward) layer. A natural extension of the theory of one-step gradient feature learning in feedforward networks suggests that one could first learn the attention weights with a single gradient step, and then train the output layer. We show that this fails for a shallow transformer trained to recognize a star-free language from unbounded-length input. We prove that the transformer can be trained to recognize the language with a three-phase gradient descent: the output layer is updated first, followed by an attention update, and the output layer is fine-tuned in the third phase. In contrast, the language cannot be learned by training only the attention or output layer, or by a single attention-layer update followed by finetuning the output layer. We empirically show that ``attention learning lagging output layer learning" is beneficial in other language recognition, regression, and in-context learning tasks as well.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20337 [cs.LG] |
| (or arXiv:2610.20337v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20337 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaomengxi Han [view email]
[v1]
Wed, 14 Oct 2026 15:19:08 UTC (817 KB)
来源：arXiv：cs.LG · arxiv.org