SlimKV：联合 token-特征 KV cache 压缩与免重建 Beacon Attention

View PDF HTML (experimental)

Abstract:Long-context LLM serving is increasingly bottlenecked by KV-cache memory, especially in resource-constrained scenarios. Among existing KV-cache compression strategies, token-wise methods reduce cached states but risk information loss through eviction or condensation, while feature-wise methods reduce per-token KV dimensions but can require full-dimensional reconstruction to apply positional embedding, limiting decoding speedups. We introduce SlimKV, a question-agnostic joint token-feature KV-cache compression method. SlimKV uses low-rank-aware training to compress long contexts into beacon memory states with latent KV representations, together with layer-adaptive rank allocation. We further uncover a positional asymmetry: removing key-side RoPE affects beacon and raw tokens differently, with much smaller degradation for beacon tokens. Exploiting this asymmetry, SlimKV trains beacon KV projections under a K-RoPE-free constraint and enables latent-space attention during decoding, mitigating reconstruction latency. On LongBench, SlimKV outperforms baselines at 16x/32x compression and remains leading at 4x/8x, where it retains over 96% of the uncompressed model's score. Needle-in-a-Haystack confirms robustness across evidence positions, and efficiency evaluation shows up to 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K length.

Comments:	Accepted to EMNLP 2026 Main Conference (Oral)
Subjects:	Machine Learning (cs.LG)
Cite as:	arXiv:2610.02953 [cs.LG]
	(or arXiv:2610.02953v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.02953 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zihan Teng [view email]
[v1] Fri, 2 Oct 2026 07:45:42 UTC (3,120 KB)