arXiv：cs.LG· Sin-Han Yang, Shih-Cheng Huang, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee·· 4 小时前AI 评分41
重新审视模型合并：任务算术引入的隐式正则化是否真的有用
A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
AI 导读
研究发现，模型合并中通过额外数据集搜索任务权重更新线性组合系数的标准做法存在隐式正则化，将候选模型限制在任务特定权重更新张成的子空间内。实验表明，去掉该正则化直接优化合并模型权重，在多种架构、领域乃至每类仅一个样本的极端数据受限场景下都显著提升常见合并方法性能，直接优化预训练权重甚至优于部分现有合并方法。分析显示更优的多任务权重存在于该子空间之外，作者呼吁重新审视现有模型合并流程。
正文
Abstract:Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. To perform well across multiple tasks, most existing merging methods use an additional dataset to find the coefficients for the best linear combination of task-specific weight updates. However, we identify an implicit regularization in this standard practice: searching over coefficients restricts the candidate models to a subspace spanned by task-specific weight updates. In this work, we investigate whether this regularization is actually useful. Surprisingly, empirical results show that optimizing merged-model weights without this regularization significantly boosts the performance of common merging methods across multiple architectures, domains, and even in an extremely data-limited scenario where only one instance is available per class. Moreover, directly optimizing the pretrained model weights even outperforms some existing merging methods. Analysis shows that better multi-task weights exist outside the subspace and can be found using multiple methods. We study different strategies for using the additional dataset, discussing their practical use and implications for model merging. Overall, this work calls for revisiting the existing model-merging pipeline, motivating a broader exploration of the weight space and a reconsideration of the implicit regularization induced by task arithmetic.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.07990 [cs.LG] |
| (or arXiv:2610.07990v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07990 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sin-Han Yang [view email]
[v1]
Tue, 6 Oct 2026 08:49:40 UTC (1,423 KB)
来源：arXiv：cs.LG · arxiv.org