arXiv：cs.LG（机器学习，全量分类）· Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu, Zihan Fang, Zhe Chen, Wei Ni, Jun Luo, Symeon Chatzinotas·· 1 天前AI 评分34
DS-MoE：通过子模优化实现依赖感知的 MoE 专家选择
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
AI 导读
研究者提出 DS-MoE 框架，将 MoE 专家选择重新定义为差子模（DS）优化问题，通过分析损失退化的二阶泰勒展开揭示专家组合中的冗余与协同双重性，并设计具有单调性保证的 MM 算法高效求解最优专家子集。实验表明 DS-MoE 能有效保留关键专家组合，性能优于现有最优基线。
正文
Abstract:While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable monotonicity guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.
| Comments: | 26 pages, 3 figures |
| Subjects: | Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2610.00558 [cs.LG] |
| (or arXiv:2610.00558v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00558 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lin Zheng [view email]
[v1]
Wed, 30 Sep 2026 18:33:27 UTC (973 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org