arXiv：cs.LG· Yehuda Dar·· 13 小时前
多零计算专家混合模型：高码率量化理论视角
Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
AI 导读
该论文用经典高码率量化理论分析回归任务中的 MoE 模型，其专家为单参数、推理时零计算的常值预测器，并假设专家数量足够大使输入空间划分区域极小。针对一维输入，论文给出了测试误差及其最优划分与专家解；针对多维输入，给出测试误差上界并研究其最小化。研究还分析了给定输入空间划分时专家参数的统计学习性质，从理论与实验上揭示了 MoE 学习中近似误差与估计误差的权衡如何取决于专家数量。
正文
Abstract:This paper uses classical high-rate quantization theory to provide new insights into mixture-of-experts (MoE) models for regression tasks. Our MoE is defined by a segmentation of the input space to regions, each with a single-parameter expert that acts as a constant predictor with zero-compute at inference. Motivated by high-rate quantization theory assumptions, we assume that the number of experts is sufficiently large to make their input-space regions very small. This lets us study the approximation error of our MoE model class: (i) for one-dimensional inputs, we formulate the test error and its minimizing segmentation and experts; (ii) for multidimensional inputs, we formulate an upper bound for the test error and study its minimization. Moreover, we consider the learning of the expert parameters from a training dataset, given an input-space segmentation, and formulate their statistical learning properties. This leads us to theoretically and empirically show how the tradeoff between approximation and estimation errors in MoE learning depends on the number of experts.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.03151 [cs.LG] |
| (or arXiv:2510.03151v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.03151 arXiv-issued DOI via DataCite |
Submission history
From: Yehuda Dar [view email]
[v1]
Fri, 3 Oct 2025 16:24:50 UTC (466 KB)
[v2]
Fri, 9 Oct 2026 17:16:42 UTC (5,872 KB)
来源：arXiv：cs.LG · arxiv.org