arXiv：cs.LG（机器学习，全量分类）· Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau·· 1 天前AI 评分34
无需数据的协方差估计实现模型合并
Model Merging via Data-Free Covariance Estimation
AI 导读
研究者提出一种无需辅助数据的模型合并方法，通过从差异矩阵直接估计逐层协方差矩阵，在保留干扰最小化框架理论依据的同时省去数据依赖并降低计算成本。该方法在视觉与语言基准上验证，覆盖 86M 至 7B 参数模型，性能超越此前无需数据的最优合并方法。
正文
Abstract:Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.01329 [cs.LG] |
| (or arXiv:2604.01329v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.01329 arXiv-issued DOI via DataCite |
Submission history
From: Marawan Gamal Abdel Hameed [view email]
[v1]
Wed, 1 Apr 2026 19:16:31 UTC (273 KB)
[v2]
Wed, 30 Sep 2026 22:54:52 UTC (278 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org