arXiv：cs.AI· Ilya Lasy, Nora Yinuo Cai, Kola Ayonrinde·· 3 小时前
RouterInterp：理解 MoE 路由中的叠加专业化现象
RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
AI 导读
针对 MoE 模型专家并非按单一领域专业化这一假设，研究者提出叠加专业化假说（SSH），认为专家实际专精于细粒度特征的并集，并据此推出路由解释方法 RouterInterp。该方法在 gpt-oss-20b 上解释专家路由的检测准确率比此前基于 token 统计的方法高出约 65%。
正文
Abstract:Sparse Mixture of Experts (MoE) models scale more efficiently than dense models by routing tokens to modular expert networks that are only active for processing a fraction of tokens. A leading hypothesis for the performance of MoE models is that each expert specialises in a single, coherent domain. However, interpretability efforts that assume this hypothesis have generally been unsuccessful. We propose and present evidence for an alternative account that we call the Superposed Specialisation Hypothesis (SSH): experts specialise in a disjoint union of fine-grained features rather than one broad domain. Leveraging the SSH, we introduce RouterInterp, a method for interpreting expert routing that identifies Sparse Autoencoder features most predictive of routing decisions and produces unified natural language explanations. On gpt-oss-20b, RouterInterp explains expert routing with ${\sim}65\%$ higher detection accuracy than prior token statistics based methods. This work provides a scalable method for generating more accurate explanations of expert routing and increases our understanding of a previously uninterpretable component of foundation models.
| Comments: | 33 pages (12 non-appendix pages), 7 figures, published as a conference paper at ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.11775 [cs.AI] |
| (or arXiv:2610.11775v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11775 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ilya Lasy [view email]
[v1]
Thu, 8 Oct 2026 11:55:25 UTC (990 KB)
来源：arXiv：cs.AI · arxiv.org