arXiv：cs.LG· Renjie Xie, Juncheng Yang, Aoting Hu, Yan Liang, Mingxi Zhang, Zheheng Hong, Wei Xu, Yanzhi Chen·· 3 小时前
NatScale：面向混合精度 MoE 量化的对比校准分配框架
Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization
AI 导读
针对混合精度 MoE 量化中专家排名无法决定误差权重的问题，研究者提出 NatScale 对比校准分配框架，用模型专属幂变换将基于梯度的专家权重校准到信息论对比目标，并结合 gate、up、down 配置的联合误差估计与权重谱先验在存储预算下选取精度配置。
正文
Abstract:Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to resilient ones, but expert rankings alone do not determine how strongly to weight their errors. To address this gap, we propose NatScale, a contrast-calibrated allocation framework for mixed-precision MoE quantization. It uses a model-specific power transformation to calibrate gradient-based expert weights to an information-theoretic contrast target while preserving rankings. We jointly evaluate each expert's gate, up and down configurations to estimate its output error. We then combine these errors and calibrated weights with an existing prior based on weight spectra, variance and precision decay to select configurations under a storage budget. Across three models, NatScale achieves the lowest native-to-quantized KL divergence and the highest top-token agreement in all nine settings against five allocation baselines. It also yields the lowest perplexity in seven settings. These findings highlight the importance of objective-aware precision allocation.
| Comments: | 26 pages, including references and appendices |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.20353 [cs.LG] |
| (or arXiv:2610.20353v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20353 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Renjie Xie [view email]
[v1]
Wed, 14 Oct 2026 15:29:40 UTC (411 KB)
来源：arXiv：cs.LG · arxiv.org