HuggingFace Daily Papers·· 1 天前
Specter：基于量化与多步投机卸载的 MoE 快速推理系统
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
AI 导读
Specter 是一个利用量化把投机解码与专家卸载结合的 MoE 推理系统，轻量量化草稿模型在设备上运行并保留高保真专家激活模式，从而对后续专家使用做多步前瞻，提前调度专家传输与执行。在单请求推理下，其 TPOT 相比 SOTA 系统平均提升 1.50x、最高 1.88x，让单 GPU 内存受限场景下的 MoE 推理更实用。该论文已被 ASPLOS 2027 接收，代码已公开。
正文 · 原文
Abstract:Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at this https URL.
| Comments: | Accepted to ASPLOS 2027. Includes the artifact appendix |
| Subjects: | Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2511.14102 [cs.LG] |
| (or arXiv:2511.14102v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2511.14102 arXiv-issued DOI via DataCite |
Submission history
From: Wang Wenfeng [view email]
[v1]
Tue, 18 Nov 2025 03:40:19 UTC (1,416 KB)
[v2]
Tue, 20 Oct 2026 14:39:33 UTC (1,627 KB)
来源：HuggingFace Daily Papers · arxiv.org