Latent Compaction：用测试时训练压缩 KV 缓存

View PDF HTML (experimental)

Abstract:Long contexts enlarge key--value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. We introduce **Latent Compaction**, which learns a compact document-specific cache through test-time training. We initialize cache entries using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed. It then jointly optimizes the entire cache through end-to-end distillation from the full-cache model. Model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate substantial gains over budget-matched compression baselines. On a RULER, for example, our method achieves 96.7% accuracy at 32$\times$ compression, compared with 40.0% for KVZip+. The method also transfers across model scales and families, with retrieval gains on both Qwen3 and MiniCPM5. These results show that treating the KV cache as a neural network and optimizing it with test-time training can preserve retrieval accuracy under aggressive compression. The code is released at this https URL

Comments:	19 pages, 21 figures
Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2610.20582 [cs.CL]
	(or arXiv:2610.20582v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.20582 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yingfa Chen [view email]
[v1] Wed, 14 Oct 2026 17:57:01 UTC (693 KB)