arXiv：cs.LG· Michela Lapenna, Rita Fioresi, Bahman Gharesifard·· 7 小时前AI 评分34
Sinkhorn 双随机注意力秩衰减分析
Sinkhorn doubly stochastic attention rank decay analysis
AI 导读
研究显示，用 Sinkhorn 算法归一化的双随机注意力矩阵比标准 softmax 行随机注意力更能保持秩，在情感分析和图像分类任务上得到验证。作者进一步推导出纯自注意力下 Sinkhorn 归一化的秩衰减理论上界：秩随深度呈双指数衰减至 1，这一现象此前已在 softmax 中被发现；与 softmax 类似，跳跃连接对缓解秩崩溃至关重要。
正文
Abstract:The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal degradation across layers. In particular, it can induce rank collapse, resulting in increasingly uniform token representations, as well as entropy collapse, characterized by highly concentrated attention distributions. Recent work has highlighted the benefits of doubly stochastic attention as a form of entropy regularization, promoting a more balanced attention distribution and leading to improved empirical performance. In this paper, we study rank collapse across network depth and show that doubly stochastic attention matrices normalized with Sinkhorn algorithm preserve rank more effectively than standard softmax row-stochastic ones. As previously shown for softmax, skip connections are crucial to mitigate rank collapse. We empirically validate this phenomenon on both sentiment analysis and image classification tasks. Moreover, we derive a theoretical bound for the pure self-attention rank decay when using Sinkhorn normalization and find that rank decays to one doubly exponentially with depth, a phenomenon that has already been shown for softmax.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC) |
| Cite as: | arXiv:2604.07925 [cs.LG] |
| (or arXiv:2604.07925v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.07925 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=fGItYoS8j1 |
Submission history
From: Michela Lapenna [view email]
[v1]
Thu, 9 Apr 2026 07:46:18 UTC (739 KB)
[v2]
Tue, 6 Oct 2026 10:16:43 UTC (765 KB)
来源：arXiv：cs.LG · arxiv.org