arXiv：cs.AI· Ali Janati, Kaoutar El Maghraoui, Chengke Zou, Xinyi Luo, Wenyuan Shen·· 6 小时前
轻量微调下的路由敏感度可识别 MoE 模型中可剪枝的专家
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
AI 导读
一种基于轻量适配器微调的路由敏感度方法可在 Mixture-of-Experts 模型中一次性剪除变化最小的专家。在 Mixtral-8×7B-Instruct 上，仅路由 LoRA 只训练 0.002% 参数，剪掉一半专家后仍保留 27.54% 的 MMLU-Pro 准确率，高于幅度剪枝和随机剪枝的约 16%，该信号随适配器增大最高升至 28.76%。
正文
Abstract:Mixture-of-Experts (MoE) models decouple parameter count from per-token compute, but deployment still requires hosting every expert in memory. Recent theory shows that experts whose router weights change least during fine-tuning can be pruned with provable accuracy preservation, yet the guarantee assumes full fine-tuning. We show that the signal can be elicited through a brief parameter-efficient adaptation. We fine-tune with a lightweight adapter, rank experts by the induced router change, and prune the least-changed experts in one shot. On Mixtral-8$\times$7B-Instruct, router-only LoRA trains 0.002% of parameters and retains 27.54% MMLU-Pro accuracy with half the experts removed, against roughly 16% for magnitude and random pruning. Signal quality improves monotonically with adapter size, reaching 28.76%, and declines as adaptation spreads beyond the router. Under their shared budget, IA3 reaches 28.04% while Houlsby reaches 25.39%. The criterion transfers to Qwen1.5-MoE fine-tuned for mathematical reasoning, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed. Structural pruning reduces memory by 49% and per-token latency by 37%. Lightweight router sensitivity therefore makes provably motivated, task-conditioned expert pruning practical at scale.
| Comments: | 26 pages, 8 figures, 12 tables. Camera-ready version accepted to AXIOM: Foundations of Efficient Deep Learning, NeurIPS 2026. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.07890 [cs.LG] |
| (or arXiv:2608.07890v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.07890 arXiv-issued DOI via DataCite |
Submission history
From: Ali Janati [view email]
[v1]
Sat, 8 Aug 2026 03:36:41 UTC (274 KB)
[v2]
Wed, 30 Sep 2026 17:50:32 UTC (436 KB)
[v3]
Sat, 10 Oct 2026 05:12:04 UTC (436 KB)
来源：arXiv：cs.AI · arxiv.org