arXiv：cs.AI· Amir Mohammad Karimi Mamaghan, Ferdinand Kapl, Andrea Dittadi, Karl Henrik Johansson, Kaitlin Maile, Johannes von Oswald, Stefan Bauer·· 1 天前
渐进深度增长如何提升自监督视觉 Transformer 的预训练效率
Growing to See: Efficient Pretraining of Self-Supervised Vision Transformers
AI 导读
研究系统考察了渐进深度增长在 MAE 和 DINOv3 两种自监督视觉 Transformer 中的效果，该方法从浅层模型开始并在训练中插入层。在全局和稠密任务上，增长策略将训练 FLOPs 减少 13.9–24.8%，ImageNet 线性探测准确率最高提升 3.36 个百分点。增长还重塑了内部计算模式：MAE 中出现全局与局部注意力交替，DINOv3 中局部到全局的注意力过渡提前。
正文
Abstract:Progressive depth growth builds large networks by starting with shallow models and inserting layers during training. Recent advances in large language models have renewed interest in this longstanding approach, demonstrating benefits for training efficiency and reasoning alongside changes in internal computations. However, how growing affects representation quality and computational patterns in self-supervised vision transformers remains unclear. In this work, we systematically investigate progressive depth growth in two popular self-supervised families, namely masked autoencoders (MAE) and DINOv3, and analyze how growing affects downstream performance, intermediate representations, and internal computations. Across global and dense tasks, we show that growing improves downstream performance while reducing training FLOPs by 13.9--24.8%, with ImageNet linear-probe accuracy gains of up to 3.36 percentage points. Growing also improves downstream performance at intermediate layers and reshapes internal computational patterns, introducing recurring alternations between global and localized attention in MAE and earlier local-to-global attention progression in self-distilled DINOv3. Together, these findings establish progressive depth growth as an effective and efficient pretraining strategy for improving downstream performance across global and dense visual tasks with less computation, and advance our understanding of how growing shapes visual computation.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.16817 [cs.CV] |
| (or arXiv:2610.16817v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16817 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Amir Mohammad Karimi Mamaghan [view email]
[v1]
Mon, 12 Oct 2026 14:01:13 UTC (4,756 KB)
来源：arXiv：cs.AI · arxiv.org