arXiv：cs.LG· Katy Blumer, Da-Cheng Juan, Jon Kleinberg, Ravi Kumar, Cyrus Rashtchian, Andrew Tomkins·· 3 小时前
Transformer 的渐近行为研究：长文本生成何时无法收敛
Asymptotic Behavior of Transformers
AI 导读
一项理论研究分析了 Transformer 长时间生成文本的渐近行为，发现即便仅三个 token 的单层纯注意力 Transformer 也可能无法收敛。
正文
Abstract:In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.
| Comments: | 24 pages, 3 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Statistics Theory (math.ST) |
| Cite as: | arXiv:2610.19358 [cs.LG] |
| (or arXiv:2610.19358v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19358 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Andrew Tomkins [view email]
[v1]
Wed, 14 Oct 2026 00:59:20 UTC (40 KB)
来源：arXiv：cs.LG · arxiv.org