TADrop：分布引导的张量级自适应稀疏化改进模型合并

Authors:Yingfeng Luo, Dingyang Lin, Junxin Wang, Kaiyan Chang, Ziqiang Xu, Weiqiao Shan, Yinqiao Li, Tong Zheng, Bei Li, Anxiang Ma, Tong Xiao, Zhengtao Yu, Jingbo Zhu

View PDF HTML (experimental)

Abstract:Merging multiple task-specific fine-tuned checkpoints into a single multi-task model provides an attractive alternative to joint multi-task training. To mitigate cross-task interference, a common strategy is to sparsify task vectors before aggregation. However, many widely used methods rely on a uniform global sparsity ratio, overlooking differences in pruning sensitivity across tasks and parameter tensors. Consequently, such uniform sparsification may remove useful parameter updates from sensitive tensors while retaining redundant updates in others. To address this limitation, we propose TADrop, a plug-and-play sparsifier that independently determines tensor-specific keep ratios from their magnitude distributions using a scale-invariant quantile ratio. TADrop operates in a data-free manner by default, while offering optional lightweight module-level calibration when a small validation set is available. Experiments across vision, language, and multimodal benchmarks, including LLM and LoRA merging, demonstrate consistent improvements across multiple host merging methods, including a 4.99-point gain over EMR-Merging on RoBERTa GLUE. In many-task ViT-B/16 merging, the improvement increases with the number of tasks, reaching 2.26 points when merging 30 tasks. Further analyses show that TADrop preserves more task-associated neurons under matched overall sparsity. Together, these findings highlight the value of distribution-guided sparsification at the tensor level for improving model merging.

Submission history

From: Yingfeng Luo [view email]
[v1] Fri, 8 Aug 2025 09:33:08 UTC (241 KB)
[v2] Tue, 13 Oct 2026 13:52:57 UTC (424 KB)