arXiv：cs.LG· Alexandre Bismuth·· 3 小时前
为什么 Graph Transformer 不在同一个 token 上聚集注意力？
Why don't Graph Transformers attend to the same token?
AI 导读
研究发现 SoTA 图 Transformer GRIT 在分子上很少形成 attention sink，且图越大 sink 越少，与 LLM 的趋势相反。原因是 GRIT 的门控 logit 评分函数：将其换成点积后，相同图上 sink 率从 0.09 升至 0.68。
正文
Abstract:In large language models (LLMs), attention concentrates heavily on the first token. This 'attention sink' is believed to protect deep models against over-mixing. Even though graph transformers face a related pressure, we find that a SoTA graph transformer, GRIT, rarely forms sinks on molecules, and that it sinks less when trained on larger graphs, the opposite of the trend in LLMs. In principle, a graph can prevent sinks: we prove and show empirically that its symmetry and locality limit the attention any node can receive. On molecules, however, with distinguishable atoms and global attention, neither limit prevents sinks; what stops them lies in the model. We find that the attention scoring function is the main cause. Replacing GRIT's gated logit with a dot product raises its sink rate from 0.09 to 0.68 on identical graphs. In Graph Transformers using dot-product attention, removing the sink gives about the same error as removing a random node, while GRIT's most-attended nodes matter to its predictions. Attention sinks are therefore a property of dot-product attention rather than of Transformers. In our experiments, they form when dot-product scoring has a distinguishable node to concentrate on.
| Comments: | Accepted at the Geometric Distributional Deep Learning (GDDL) workshop at NeurIPS 2026. 8 pages, 1 figure. Code: this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19324 [cs.LG] |
| (or arXiv:2610.19324v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19324 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alexandre Bismuth [view email]
[v1]
Wed, 14 Oct 2026 00:02:42 UTC (51 KB)
来源：arXiv：cs.LG · arxiv.org