arXiv：cs.AI· Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, Yilun Yao, Tong Yang·· 5 小时前
Prediction Is All MoE Needs：专家负载分布从波动走向稳定
Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
AI 导读
研究追踪多个大语言模型训练中每个专家的负载，将专家负载划分为"明显波动"的瞬态和具备"时间局部性"的稳定态，并部署三种经典预测算法实现准确的专家负载预测。在 GPT3 350M 模型上，预测未来 1,000 步和 2,000 步专家负载比例的平均误差率约为 1.3% 和 1.8%。该工作可为 MoE 模型训练的专家放置与资源分配提供指导。
正文
Abstract:MoE facilitates the development of large models by making the computational complexity of the model no longer scale linearly with increasing parameters. The learning sparse gating network selects a set of experts for each token to be processed; however, this may lead to differences in the number of tokens processed by each expert over several successive iterations, i.e., the expert load fluctuations, which reduces computational parallelization and resource utilization. To this end, we traced and analyzed loads of each expert in the training iterations for several large language models in this work, and defined the transient state with "obvious load fluctuation" and the stable state with "temporal locality". Moreover, given the characteristics of these two states and the computational overhead, we deployed three classical prediction algorithms that achieve accurate expert load prediction results. For the GPT3 350M model, the average error rates for predicting the expert load proportion over the next 1,000 and 2,000 steps are approximately 1.3% and 1.8%, respectively. This work can provide valuable guidance for expert placement or resource allocation for MoE model training. Based on this work, we will propose an expert placement scheme for transient and stable states in our coming work.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2404.16914 [cs.LG] |
| (or arXiv:2404.16914v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2404.16914 arXiv-issued DOI via DataCite |
Submission history
From: Peizhuang Cong [view email]
[v1]
Thu, 25 Apr 2024 15:39:59 UTC (3,545 KB)
[v2]
Sat, 10 Oct 2026 08:15:18 UTC (3,544 KB)
来源：arXiv：cs.AI · arxiv.org