arXiv：cs.LG· Mingyuan Zhang, Yue Bai, Huan Wang, Yizhou Wang, Qihua Dong, Yitian Zhang, Yun Fu·· 6 小时前AI 评分35
Mask Fine-Tuning：通过二值掩码微调提升大语言模型性能
Boosting Large Language Models with Mask Fine-Tuning
AI 导读
研究者提出 Mask Fine-Tuning（MFT），一种在不更新模型权重的前提下，通过学习并应用二值掩码来提升 LLM 性能的微调新范式。在 LLaMA2-7B/3.1-8B 上，MFT 使用相同微调数据集在 IFEval 上实现平均 2.70/4.15 的提升，且跨领域与骨干模型保持一致增益。该方法可与其他 LLM 优化流程兼容，并将掩码操作从网络剪枝拓展到更广泛的模型能力范畴。
正文
Abstract:The large language model (LLM) is typically integrated into the mainstream optimization protocol. However, it remains underexplored whether maintaining the model integrity is \textit{indispensable} for promising performance. In this work, we introduce Mask Fine-Tuning (MFT), a novel LLM fine-tuning paradigm demonstrating that carefully breaking the model's structural integrity can surprisingly improve performance without updating model weights. MFT learns and applies binary masks to well-optimized models, using the standard LLM fine-tuning objective as supervision. Based on fully fine-tuned models, MFT uses the same fine-tuning datasets to achieve consistent performance gains across domains and backbones (e.g., an average gain of 2.70/4.15 on IFEval with LLaMA2-7B/3.1-8B). Detailed ablation studies and analyses examine the proposed MFT from different perspectives, including the sparse ratio and the loss surface. Additionally, when deployed on well-trained models, MFT is compatible with other LLM optimization procedures to improve overall model performance. Furthermore, this study extends the masking operation beyond its conventional use in network pruning for model compression to encompass a broader range of model capabilities.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2503.22764 [cs.CL] |
| (or arXiv:2503.22764v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2503.22764 arXiv-issued DOI via DataCite |
Submission history
From: Yue Bai [view email]
[v1]
Thu, 27 Mar 2025 20:17:57 UTC (3,119 KB)
[v2]
Sun, 15 Mar 2026 11:21:40 UTC (3,654 KB)
[v3]
Tue, 6 Oct 2026 06:26:46 UTC (3,218 KB)
来源：arXiv：cs.LG · arxiv.org