arXiv：cs.AI· Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang·· 16 小时前
SpecFit：MoE 推理卸载中并非所有草稿 token 代价相同
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
AI 导读
针对 MoE 模型专家权重卸载到主机内存时 H2D 传输拖慢解码的问题，研究者提出 SpecFit，按预测吞吐量而非接受长度选择已验证的草稿子树。在 CNN/DailyMail 摘要和 GSM8K 子集上，Qwen3-30B-A3B 使用相同专家缓存时，SpecFit 解码速度比 EAGLE-3 快 21.6% 和 7.0%，是原版 EAGLE-3 的 1.97x 和 1.72x。代码已公开。
正文
Abstract:Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15417 [cs.AI] |
| (or arXiv:2610.15417v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15417 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiuyu Li [view email]
[v1]
Sun, 11 Oct 2026 07:41:12 UTC (55 KB)
来源：arXiv：cs.AI · arxiv.org