arXiv：cs.LG· Yuping Lin, Jiayuan Ding, Yue Xing, Pengfei He, Jiliang Tang, Subhabrata Mukherjee·· 2 小时前
VECTOR：一种改进基于驱逐的 KV Cache 压缩的简易插件
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
AI 导读
VECTOR 是一种用于驱逐式 KV Cache 压缩流程的即插即用增强方法，引入保留、近似、驱逐三路 token 路由，结合基础评分器的重要性信号与离线校准回归价值估计的可重建性信号。
正文
Abstract:KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction or representation approximation, which may underutilize tokens that are not critical for exact retention but are still reconstructable. We present VECTOR, a plug-and-play augmentation for eviction-based pipelines that introduces three-way token routing: retention, approximation, and eviction. VECTOR combines an importance signal from the base scorer with a reconstructability signal from an offline-calibrated regression-based value estimation. By leveraging reconstructability, VECTOR recovers useful value information that would otherwise be irreversibly lost under binary eviction, while preserving key vectors for attention routing stability. On LongBench, RULER with contexts up to 128K tokens, and multi-turn SCBench, VECTOR yields significant gains over its base eviction methods under high compression. At a 90% compression ratio, SnapKV augmented with VECTOR outperforms six competing compression methods, including D2O and RocketKV, on all three evaluated models. Since VECTOR makes all routing decisions at prefill, decoding runs on a standard compressed cache with no measurable overhead.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.23258 [cs.LG] |
| (or arXiv:2605.23258v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23258 arXiv-issued DOI via DataCite |
Submission history
From: Yuping Lin [view email]
[v1]
Fri, 22 May 2026 06:00:15 UTC (1,401 KB)
[v2]
Tue, 13 Oct 2026 19:50:35 UTC (1,424 KB)
来源：arXiv：cs.LG · arxiv.org