arXiv：cs.LG· Kaushik Pendiyala, Haris Zia, Trevin Lee, Timothy Legge, Alejandro J. De Leon, Zihan Zhao, Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Richard Cavanaugh, Javier Duarte·· 3 小时前AI 评分30
MoE Particle Transformer 的条件容量与路由研究
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
AI 导读
研究在 188 类 JetClass-II 上对比稠密与 MoE Particle Transformer，发现避免 token dropping 时 top-1 MoE 在近乎不变的标称前向计算下优于稠密基线，继续增加存储专家数量带来的精度提升很小。每 token 激活多个专家可进一步提升预测性能，但计算成本更高；路由分析显示专家分配与粒子身份和运动学特征的关联并不随分类性能单调增强。
正文
Authors:Kaushik Pendiyala, Haris Zia, Trevin Lee, Timothy Legge, Alejandro J. De Leon, Zihan Zhao, Aaron Wang, Abhijith Gandrakota, Jennifer Ngadiuba, Richard Cavanaugh, Javier Duarte
Abstract:Mixture-of-Experts (MoE) models can increase parameter capacity without proportionally increasing active computation, but it is unclear how this trade-off behaves in particle-physics transformers. We study dense and MoE Particle Transformers on 188-class JetClass-II, varying expert count, routing capacity, top-K, and auxiliary loss. We find that, when token dropping is avoided, top-1 MoE models improve over the dense baseline at nearly unchanged nominal forward compute, while further increasing the number of stored experts produces little additional accuracy gain. Activating multiple experts per token yields additional predictive improvements at higher computational cost. Routing analyses show that expert assignments become more strongly associated with particle identity and kinematics in some configurations, but this structure does not increase monotonically with classification performance. These results highlight the need to distinguish stored parameter capacity, active computation, routing capacity, and routing organization when evaluating sparse expert models for jet classification. Code and experiment configurations are available at this https URL.
| Comments: | 8 pages, 4 figures. Submitted to the ML4PS 2026 workshop |
| Subjects: | Machine Learning (cs.LG); High Energy Physics - Experiment (hep-ex) |
| Report number: | FERMILAB-PUB-26-0717-CMS-LDRD |
| Cite as: | arXiv:2610.02701 [cs.LG] |
| (or arXiv:2610.02701v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02701 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kaushik Pendiyala [view email]
[v1]
Fri, 2 Oct 2026 02:35:28 UTC (533 KB)
来源：arXiv：cs.LG · arxiv.org