arXiv：cs.AI· Longhua Li, Lei Qi, Qi Tian, Xin Geng·· 1 天前
面向多任务知识聚合的 essential subspace 数据感知模型合并
Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregation
AI 导读
研究提出数据感知的模型合并框架 ESD、ESM 与 ESM++，用于聚合共享 checkpoint 微调模型的多任务知识。ESD 依据 proxy data 上的激活偏移识别每个任务的 essential subspace，ESM 按输出偏移特征值加权任务更新方向并跨任务正交化以减少干扰，ESM++ 进一步动态集成低秩任务残差专家。
正文
Abstract:Model merging enables training-free multi-task learning by aggregating knowledge from models fine-tuned from a shared checkpoint, but their task updates can interfere. Using proxy data, we find that task updates induce output shifts whose energy concentrates in a few dominant directions, while many low-energy directions can accumulate across tasks and impair performance. We call the span of the dominant directions the essential subspace. This observation motivates a data-aware framework to preserve essential task knowledge while reducing interference. Essential Subspace Decomposition (ESD) identifies each task's essential subspace from activation shifts measured on proxy data and represents its update compactly while preserving functional behavior. Essential Subspace Merging (ESM) weights each task-update direction by its corresponding output-shift eigenvalue, thereby prioritizing core task knowledge; it then orthogonalizes the weighted directions across tasks to reduce interference. ESM++ extends this merged model by dynamically integrating low-rank task-specific residual experts for adaptive knowledge aggregation. Experiments across diverse task sets and model scales show that ESM and ESM++ effectively aggregate task knowledge while mitigating inter-task interference. Code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.19164 [cs.LG] |
| (or arXiv:2606.19164v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.19164 arXiv-issued DOI via DataCite |
Submission history
From: Longhua Li [view email]
[v1]
Wed, 17 Jun 2026 15:07:11 UTC (4,289 KB)
[v2]
Sat, 20 Jun 2026 01:44:55 UTC (4,282 KB)
[v3]
Tue, 23 Jun 2026 03:46:45 UTC (4,278 KB)
[v4]
Tue, 13 Oct 2026 13:09:12 UTC (2,416 KB)
来源：arXiv：cs.AI · arxiv.org