arXiv：cs.LG· Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh, Vahid Partovi Nia, Parsa Omidi·· 11 小时前
Kronecker Product Attention：通过结构化分解实现内存高效 KV 压缩
Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization
AI 导读
研究提出 Kronecker Product Attention（KPA），将 key 和 value 激活表示为紧凑、token 相关矩阵因子的 Kronecker 积之和，最高实现 10x KV cache 压缩。
正文
Abstract:Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size, and model depth. Existing approaches such as Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Tensor Product Attention (TPA) reduce this cost through different trade-offs, but often sacrifice expressivity, require additional positional-encoding machinery, or impose restrictive low-rank structure. We propose Kronecker Product Attention (KPA), which represents key and value activations as sums of Kronecker products of compact, token-dependent matrix factors. KPA strictly generalizes TPA: rank-one vector-outer-product factorization arises as a degenerate Kronecker split, while non-degenerate matrix factors can represent richer block-structured patterns. We further show that a structured rotary embedding operator matched to the Kronecker representation preserves the relative-position property of RoPE while rotating only the compact factors, removing the need for a decoupled RoPE branch as in MLA. Empirically, KPA matches or exceeds TPA, MHA, MQA, GQA, and MLA across four model scales and nine downstream benchmarks under identical training conditions, while achieving up to 10x KV cache compression. The Kronecker head split is a tuneable hyperparameter that allows practitioners to navigate the compression-accuracy trade-off at design time without architectural changes.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.12657 [cs.LG] |
| (or arXiv:2610.12657v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12657 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bahareh Nikpour [view email]
[v1]
Thu, 8 Oct 2026 19:23:56 UTC (227 KB)
来源：arXiv：cs.LG · arxiv.org