arXiv：cs.LG· Maximilian Schambach, Clemens Biehl, Sam Thelin·· 13 小时前
表格基础模型的注意力机制基准测试：Torch SDPA、FlashAttention 与 vLLM 等后端对比
Benchmarking Attention for Tabular Foundation Models
AI 导读
研究团队为 TabPFN、Mitra、ConTextTab 等表格上下文学习模型搭建了可复现的注意力基准测试，在 A100、H100、B200 三代 GPU 上对比 Torch SDPA、FlashAttention-2/3/4、vLLM 和 SageAttention 等后端的前向与反向吞吐。
正文
Abstract:Tabular in-context learners such as TabPFN, Mitra, or ConTextTab rely on alternating row and column attention over 2D sequences of latent embeddings. These attention patterns differ markedly from the one-dimensional case in language models: row attention involves longer sequences while column attention operates on much shorter ones, and the strided memory layout of tabular data makes producing contiguous tensors costly. Moreover, the hidden dimensions used in current models are small compared to recent language models. Yet efficient attention has been studied mostly for one-dimensional sequences, leaving the two-dimensional tabular setting unexplored. To this end, we create a reproducible benchmarking setup and study the unique characteristics of tabular attention across several backends -- Torch SDPA (efficient and cuDNN), FlashAttention-2/3/4, and the inference-only backends vLLM and SageAttention -- measuring forward and backward throughput across realistic tabular shapes on three GPU generations (A100, H100, B200). We find that the optimal backend choice differs between column and row attention and varies across hardware as well as model specifics: While the FlashAttention implementations tailored for each GPU generation perform overall best, they are at times outperformed by CuDNN in the case of column attention at longer sequences with cross-over points depending on the head dimension. Among inference-only backends, SageAttention performs well for row attention and large sequences beyond 16\,k rows. Our reproducible benchmark lays the foundation for future improvements to table-native attention. The self-contained benchmarking and evaluation code is openly available at: this https URL
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.31306 [cs.LG] |
| (or arXiv:2609.31306v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.31306 arXiv-issued DOI via DataCite |
Submission history
From: Maximilian Schambach [view email]
[v1]
Fri, 25 Sep 2026 14:14:48 UTC (480 KB)
[v2]
Mon, 28 Sep 2026 10:55:52 UTC (480 KB)
[v3]
Fri, 9 Oct 2026 12:24:48 UTC (998 KB)
来源：arXiv：cs.LG · arxiv.org