arXiv：cs.LG· Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo·· 2 小时前
多语言 MoE 如何随语言覆盖范围改变容量与算力分配
Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation
AI 导读
一项研究提出重复感知的联合缩放定律，将活跃参数量、各语言训练 token 数、专家数量与语言覆盖范围关联起来。该定律基于 64 个预训练配置、26,300 条单语言损失观测值拟合，对留出配置的预测误差比最强基线降低 28%。在活跃参数量和各语言训练 token 数匹配时，32 个专家将语言从 12 种扩展到 200 种的损失代价较稠密模型降低 83%。
正文
Abstract:Multilingual models share capacity across languages, so adding languages can hurt performance on existing ones. Mixture-of-experts models expand this capacity with little additional per-token computation. Given this extra capacity, does adding experts reduce what each additional language costs? To answer this question, we propose a repetition-aware joint scaling law relating number of active parameters, training tokens per language, number of experts, and language coverage. Fitted on 26,300 per-language loss observations from 64 pretrained configurations, our law reduces prediction error on held-out configurations by 28% relative to the strongest evaluated baseline. At matched number of active parameters and training tokens per language, 32 experts reduce the loss penalty of expanding from 12 to 200 languages by 83% compared with a dense model. Our law further shows that broader language coverage favors more experts across a wider range of compute budgets, guiding capacity allocation as multilingual models scale.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20247 [cs.CL] |
| (or arXiv:2610.20247v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nikit Srivastava [view email]
[v1]
Wed, 14 Oct 2026 14:20:19 UTC (248 KB)
来源：arXiv：cs.LG · arxiv.org