arXiv：cs.LG· Huaiyuan Qin, Muli Yang, Gabriel James Goenawan, Peng Hu, Chen Gong, Xi Peng, Hongyuan Zhu·· 4 小时前
研究发现：Attention Transfer 对 Vision Transformer 并非普遍有效
Attention Transfer Is Not Universally Effective for Vision Transformers
AI 导读
一项在 11 个 ViT 家族、20 个教师模型上的基准测试发现，Attention Transfer 并非普遍有效：7 个家族迁移成功，4 个家族持续失败，最低比从头训练的无迁移基线低 5.1%。
正文
Abstract:A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard student ViT, is sufficient to recover the full benefit of the teacher's pre-trained weights. We revisit this finding on a comprehensive benchmark of 20 teachers from 11 well-known ViT families and reveal that Attention Transfer is not universally effective. While 7 families transfer successfully, 4 consistently fail, falling up to 5.1\% below the from-scratch no-transfer baseline. Further results demonstrate that this failure is family-consistent across model sizes, and persists under extended training durations, different transfer datasets, and out-of-distribution evaluations. Controlled analyses then consistently localize the problem to the attention-routing channel, indicating that the key issue is not whether the student can match the teacher's attention patterns, but whether the matched patterns remain functional for the student. Crucially, we identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism. By adding only the teacher's native architectural components to the student in a randomly initialized state, we completely reverse the failure for all 4 families. Notably, these components alone do not improve from-scratch training, confirming that they specifically unlock the usability of the teacher's attention. We further show that this failure is not explained by the inadequate choice of transfer loss, differences in pre-training recipes, or the intrinsic quality of attention maps. Our findings refine the prevailing understanding of attention in ViT representations: attention is sufficient \textit{only} when the student architecture matches the teacher.
| Comments: | Published in NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.07191 [cs.CV] |
| (or arXiv:2605.07191v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07191 arXiv-issued DOI via DataCite |
Submission history
From: Huaiyuan Qin [view email]
[v1]
Fri, 8 May 2026 03:39:38 UTC (227 KB)
[v2]
Wed, 14 Oct 2026 13:35:30 UTC (227 KB)
来源：arXiv：cs.LG · arxiv.org