arXiv：cs.LG· Mingyuan Zhang, Yue Bai, Yifan Wang, Yiyang Huang, Yun Fu·· 6 小时前AI 评分35
重新思考微调：Mask Fine-Tuning（MFT）无需改动权重即可释放视觉语言模型隐藏能力
Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
AI 导读
研究者提出 Mask Fine-Tuning（MFT），一种不改动 VLM 主干权重、通过掩码选择性路由预训练连接的结构化适配方法，替代 FFT 与 PEFT。在多个视觉语言基准上，MFT 持续取得更优性能，且不新增知识、不改变部署架构。该工作还揭示预训练 VLM 在适配过程中内部表征路径的重组方式。
正文
Abstract:Fine-tuning has become the dominant paradigm for adapting Vision-Language Models (VLMs), yet most approaches rely on explicit weight updates that introduce a fundamental trade-off. Full Fine-Tuning (FFT) may perturb pretrained representations due to cross-modal gradient interference, whereas Parameter-Efficient Fine-Tuning (PEFT) methods rely on additive modules, such as low-rank adapters, which may limit adaptation capacity. In this paper, we rethink VLM adaptation from a structural selection framework that adapts VLMs without modifying backbone weights, and we propose Mask Fine-Tuning (MFT). MFT learns masks that selectively route information through existing pretrained connections, dynamically uncovering subnetworks that better align pretrained representations with downstream objectives. Extensive experiments show that MFT provides an effective structural alternative to both FFT and PEFT, consistently achieving superior performance across multiple vision-language benchmarks without adding knowledge or altering the deployment architecture. Moreover, our analysis with MFT provides new insights into how pretrained VLMs reorganize their internal representational pathways during adaptation.
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2512.23073 [cs.LG] |
| (or arXiv:2512.23073v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.23073 arXiv-issued DOI via DataCite |
Submission history
From: Mingyuan Zhang [view email]
[v1]
Sun, 28 Dec 2025 20:41:22 UTC (2,161 KB)
[v2]
Tue, 6 Oct 2026 06:16:15 UTC (2,968 KB)
来源：arXiv：cs.LG · arxiv.org