arXiv：cs.AI· Yaoxiang Wang, Kai Zhang, Ruobing Xie, Lanrui Wang, Guoliang Zhao, Junhui Li, Jinsong Su·· 7 小时前
LatentMoE：从专家几何到逐层容量分配的 MoE 设计
Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation
AI 导读
LatentMoE 将专家计算移入低维空间，使专家的输入维度与中间宽度成为显式设计变量；专家局部 Jacobian 秩由这两者共同约束，在单专家参数量固定时二者均衡可最大化秩上限，实验显示接近方形的专家更优。该方法还用逐层内在维度（ID）指导深度上的容量分配，在每层参数量、激活 FLOPs 与通信量不变的前提下协同调整专家数量与形状，相比逐层同构基线取得更低验证损失和更强下游表现。
正文
Abstract:Fine-grained mixture-of-experts (MoE) models commonly increase expert count by narrowing each expert, treating expert shape as a by-product of routing granularity. LatentMoE moves expert computation into a lower-dimensional space, exposing the input dimension and intermediate width of each expert as explicit design choices. We show that an expert's local Jacobian rank is jointly bounded by these two dimensions, and that balancing them maximizes the rank ceiling at fixed per-expert parameters. Controlled shape sweeps across model scales and routing granularities consistently favor near-square experts. We further use layerwise intrinsic dimension (ID) to guide capacity allocation across depth, co-varying the number and shape of experts while keeping per-layer parameter count, activated FLOPs, and communication volume constant. Compared with layer-homogeneous baselines, our ID-guided allocation achieves lower validation loss and stronger downstream performance. These results establish expert shape and layerwise allocation as practical design axes for MoEs.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14644 [cs.AI] |
| (or arXiv:2610.14644v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14644 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaoxiang Wang [view email]
[v1]
Sat, 10 Oct 2026 06:54:36 UTC (199 KB)
来源：arXiv：cs.AI · arxiv.org