arXiv：cs.LG· Mingyuan Zhang, Yue Bai, Zhongruo Wang, Yupin Huang, Yiyang Huang, Hailing Wang, Huimin Zeng, Yun Fu·· 4 小时前AI 评分35
MASKerade：面向稠密到 MoE 升级的 Token 路由掩码专家
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
AI 导读
MASKerade 是一种稠密到 MoE 升级训练方法，将专家学习为冻结预训练 FFN 的稀疏子网络，每个专家由学习到的二值掩码定义，token 级路由器选择并组合要执行的掩码 FFN。路由器与掩码分数联合优化，底层 FFN 权重保持不变。主配置采用四个 2:4 专家与 top-2 路由，在 Qwen 和 Gemma 骨干的五个视觉语言基准上取得对比基线中最高性能。
正文
Abstract:Sparsely activated Mixture-of-Experts (MoE) models increase model capacity without a proportional increase in per-token computation. Dense-to-MoE upcycling reuses pretrained dense models to construct such systems, commonly by copying feed-forward networks (FFNs) into independently trained experts. We introduce MASKerade, a dense-to-MoE training method that instead learns experts as sparse subnetworks of a frozen pretrained FFN. Each expert is defined by a learned binary mask, and a token-level router selects which masked FFNs to execute and combine. The router and mask scores are optimized jointly, while the underlying FFN weight values remain unchanged. This formulation supports neuron-structured, semi-structured, and unstructured experts within the same routing architecture. Our main configuration uses four 2:4 experts with top-2 routing, where two half-dense expert passes have the nominal FFN arithmetic of one dense pass, without requiring independent expert weight matrices. On five vision-language benchmarks with Qwen and Gemma backbones, this configuration achieves the highest performance among the compared baselines. Comparisons across mask granularities, routing interventions, and compute-matched controls distinguish the effects of learned connectivity from expert activation count. These results establish mask learning over frozen weights as a practical alternative for constructing token-routed MoE experts.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.07809 [cs.LG] |
| (or arXiv:2610.07809v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07809 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mingyuan Zhang [view email]
[v1]
Tue, 6 Oct 2026 06:04:38 UTC (820 KB)
来源：arXiv：cs.LG · arxiv.org