TreeGraft：面向树状投机解码的自适应多草稿模型嫁接框架

View PDF HTML (experimental)

Abstract:Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at this https URL.

Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2608.26112 [cs.CL]
	(or arXiv:2608.26112v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2608.26112 arXiv-issued DOI via DataCite

Submission history

From: Jiaming Fan [view email]
[v1] Thu, 28 May 2026 17:52:11 UTC (620 KB)
[v2] Fri, 28 Aug 2026 04:00:32 UTC (620 KB)
[v3] Wed, 14 Oct 2026 12:28:01 UTC (648 KB)