arXiv：cs.LG（机器学习，全量分类）· Sibylle Marcotte, Joan Bruna·· 14 小时前AI 评分40
深度残差自注意力网络的通用插值研究
Universal interpolation for deep residual self-attention networks
AI 导读
一项研究证明，仅用两个冻结的单头残差 softmax 注意力块（投影矩阵为高斯初始化），即可将任意 N 条含 n 个 token 的序列集合映射到任意另一集合，参数完全固定，仅块的应用顺序、符号和持续时间随任务变化。该结果在连续深度和有限深度下均成立，并给出了因果掩码下的相应通用插值保证。
正文
Abstract:Universal approximation is a necessary qualitative property of learning architectures to benefit from scaling laws. While it is generically verified on a variety of neural architectures and random feature models, it typically involves infinite width limits. In this work, we focus on deep self-attention models and consider instead the `dual' regime, where approximation power is enabled entirely by depth, and featuring strong parameter sharing across layers, motivated by recent models such as the Looped Transformers. More specifically, we ask whether one can find a predefined finite set of parameters, each defining an attention block, such that the resulting finite set of transformations can map any collection of $N$ sequences of $n$ tokens to any other collection of $N$ sequences of $n$ tokens. Crucially, these transformations are \emph{fixed independently of the input and output} collections: only the order in which the blocks are applied, their signs, and their durations depend on the particular interpolation task. Our main result establishes it for residual softmax attention using only two frozen single-head blocks with Gaussian-initialized projection matrices. The result holds at both continuous and finite depth. We also characterize the restrictions imposed by causal masking and establish corresponding universal interpolation guarantees.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.01981 [cs.LG] |
| (or arXiv:2610.01981v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.01981 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sibylle Marcotte [view email]
[v1]
Thu, 1 Oct 2026 16:25:13 UTC (98 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org