arXiv：cs.LG· Amir Mohammad Mahfoozi, Zi Yang, Ying Li, Michael Minyi Zhang·· 4 小时前AI 评分37
RFF-GPA：线性时间复杂度的随机特征高斯过程注意力，实现校准不确定性
Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty
AI 导读
研究者提出即插即用的随机傅里叶特征高斯过程注意力（RFF-GPA）模块，将注意力建模为用随机傅里叶特征近似平稳核的高斯过程，把后验均值与方差的近似复杂度从三次或二次降至序列长度的线性时间。在多个真实数据集上，该模块在保持预测精度的同时改善了校准效果。
正文
Abstract:Transformers provide a state-of-the-art modeling framework, yet poor calibration limits their reliability in safety-critical applications. A promising direction addresses this issue by interpreting attention as a Gaussian process (GP) posterior, which enables principled uncertainty calibration but incurs cubic complexity in sequence length due to the inversion of the kernel; although decoupled GP variants reduced the cost to quadratic, the computation remains prohibitive in practice. In this paper, we propose the plug-and-play random Fourier feature Gaussian process attention (RFF-GPA) module, which represents the attention as a GP with a stationary kernel approximated by random Fourier features. This low-rank approximation results in linear-time complexity for approximating the posterior mean and variance, making it far more scalable compared to previous work. Empirical results on multiple real-world datasets show that our attention module improves calibration while maintaining predictive accuracy, and simultaneously reduces computational complexity to linear in the sequence length.
| Comments: | 14 pages, 3 figures, 3 tables |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.08578 [cs.LG] |
| (or arXiv:2610.08578v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08578 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amir Mohammad Mahfoozi [view email]
[v1]
Tue, 6 Oct 2026 15:51:29 UTC (1,075 KB)
来源：arXiv：cs.LG · arxiv.org