arXiv：cs.LG· Gil Jae Song, Sungjoo Yoo·· 4 小时前
PN-Mix：面向长输出 LLM 推理的配对范数引导混合精度 KV Cache 量化
PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
AI 导读
PN-Mix 是一种面向 RoPE 解码器 LLM 的混合精度 key cache 量化方法，通过 query-key 配对范数乘积识别显著通道对，仅在 prefill 阶段计算一次并在整个解码过程复用。
正文
Abstract:Long-output inference in large language models suffers from severe KV cache memory and bandwidth bottlenecks. While KV cache quantization alleviates this issue, low-bit quantization (e.g., 2-bit) severely degrades model accuracy. Recent mixed-precision key-cache quantization methods mitigate this degradation by allocating higher precision to important key channels, but estimate channel saliency using only key statistics. Through attention logit perturbation analysis at the RoPE channel-pair level, we show that quantization errors in channel pairs with large query--key pair-norm products contribute significantly to logit variation. Based on this insight, we propose PN-Mix, a mixed-precision key-cache quantization method for RoPE-based decoder-only LLMs, which allocates higher precision to salient channel pairs identified once during the prefill stage and reused throughout decoding. Across standard reasoning, code-generation, and AIME benchmarks, PN-Mix with 2.26-bit average precision achieves comparable or better average accuracy than prior mixed-precision methods using 2.32--2.39 bits. Furthermore, in our Pareto sweep, PN-Mix degrades gracefully down to 1.5 bits, whereas the compared baselines exhibit sharper performance drops below 2.25 bits.
| Comments: | EMNLP 2026 |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19605 [cs.LG] |
| (or arXiv:2610.19605v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19605 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gil Jae Song [view email]
[v1]
Wed, 14 Oct 2026 05:29:08 UTC (874 KB)
来源：arXiv：cs.LG · arxiv.org