arXiv：cs.LG· Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye·· 4 小时前AI 评分44
SchemaFill：通过槽位并行推测解码实现高效 LLM 工具调用
SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding
AI 导读
SchemaFill 是一个通过槽位并行推测解码实现高效 LLM 工具调用的框架，可并行生成未来槽位值作为候选，再由目标模型在实际输出前缀下验证，仅提交通过验证的 token。在 Glaive 和 BFCL 上，SchemaFill 相比自回归解码实现最高 4.05 倍的端到端吞吐量提升，代码已开源。
正文
Abstract:LLM agents interact with external systems by generating structured tool calls. Given a user request, conversational context, and a catalog of tool schemas, a tool-calling model must select tools and generate their arguments, potentially producing multiple calls in a single response. Standard autoregressive decoding generates these calls token by token, incurring substantial latency for requests involving multiple calls or many argument fields. The explicit argument structure offers opportunities for parallel generation, but later argument values may depend on preceding fields and calls, so independently generated values can differ from the target model's output. We present SchemaFill, a framework for efficient LLM tool calling through slot-parallel speculative decoding. SchemaFill generates future slot values concurrently as candidates, without requiring advance knowledge of the actual call sequence or argument values. Candidates spanning multiple fields and calls are concatenated for verification by the target model under the actual output prefix. Only verified tokens are committed, and the target supplies corrections when candidates disagree. This applies target verification while exploiting parallelism across slots and calls. On Glaive and BFCL, SchemaFill achieves up to a 4.05$\times$ improvement in end-to-end throughput over autoregressive decoding. Code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07086 [cs.LG] |
| (or arXiv:2610.07086v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07086 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhikai Chen [view email]
[v1]
Mon, 5 Oct 2026 12:22:34 UTC (1,388 KB)
来源：arXiv：cs.LG · arxiv.org