arXiv：cs.LG· Simon Gabet (LMO), Etienne Boursier (LMO, CELESTE), Claire Boyer (LMO, IUF)·· 4 小时前
高斯混合上的 Softmax Attention：能线性时线性，必须选择时选择
Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
AI 导读
这项理论研究分析了高斯混合数据上 softmax attention 的无限提示词极限，发现它既能像线性 attention 一样有效恢复线性任务，又能利用依赖查询的上下文选择解决线性 attention 无法处理的非线性任务。研究通过基于梯度的方法证明，softmax attention 可表示并学习监督分类和去噪等统计任务的最优解。
正文
Abstract:Softmax attention, at the heart of Transformers, has demonstrated remarkable capabilities. Yet its underlying mechanisms remain only partially understood. Recent theoretical work studies Gaussian prompts, where the infinite-prompt limit reduces softmax attention to a linear map, but also removes the query-dependent selection that distinguishes it from linear attention. This work studies the infinite-prompt limit of softmax attention on Gaussian mixtures, which retain the tractability of Gaussian data while introducing latent structure, multimodality, and nonlinear dependencies. We show that softmax attention can represent and learn, via gradient-based methods, optimal solutions to a range of statistical tasks, including supervised classification and denoising. Our results highlight two complementary capabilities of softmax attention: it can recover linear tasks as effectively as its simpler linear counterpart, while also exploiting query-dependent context selection to solve nonlinear tasks beyond the reach of linear attention.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11798 [stat.ML] |
| (or arXiv:2610.11798v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11798 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Simon Gabet [view email] [via CCSD proxy]
[v1]
Thu, 8 Oct 2026 12:09:57 UTC (432 KB)
来源：arXiv：cs.LG · arxiv.org