arXiv：cs.LG· Frank Cole, Nicholas H. Nelsen, Takashi Furuya·· 4 小时前AI 评分27
Transformer 在次高斯数据上的测度到测度变换稳定性研究
Stability of Measure-to-Measure Transformers on Sub-Gaussian Data
AI 导读
一项数学研究证明 Transformer 将次高斯输入映射为次高斯输出，从而保证 softmax 算子任意长度复合的良定义性；研究进一步证明 Transformer 关于 1-Wasserstein 距离满足 Hölder 连续性，并据此给出次高斯输入与其经验近似之间的误差传播估计。
正文
Abstract:Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are Hölder continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different Hölder regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.07717 [stat.ML] |
| (or arXiv:2610.07717v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07717 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Frank Cole [view email]
[v1]
Tue, 6 Oct 2026 04:13:29 UTC (70 KB)
来源：arXiv：cs.LG · arxiv.org