arXiv：cs.CL· Taiga Ikeda, Daisuke Miyashita, Jun Deguchi·· 13 小时前
KV-Repurpose：面向非前缀 KV-Cache 复用的评测平台
KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
AI 导读
vLLM 开源系统 KV-Repurpose 用于开发和评测超越精确前缀的 segment 级 KV-cache 复用，结合缓存感知段调度、提示词构建、容量管理与运行时监控。
正文
Abstract:KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes. The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning. Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT. Even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload. Source code is available under the Apache-2.0 license at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13060 [cs.CL] |
| (or arXiv:2610.13060v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13060 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Taiga Ikeda [view email]
[v1]
Fri, 9 Oct 2026 06:17:10 UTC (844 KB)
来源：arXiv：cs.CL · arxiv.org