arXiv：cs.LG· Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong·· 3 小时前
OScaR：面向 LLM 及更多模型的极致 KV Cache 量化"奥卡姆剃刀"
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
AI 导读
针对长上下文与多模态推理中 KV cache 显存瓶颈，研究者提出轻量压缩框架 OScaR（Omni-Scaled Canalized Rotation），将量化失真主因归结为 Token Norm Imbalance（TNI），通过 Canalized Rotation 与 Omni-Token Scaling 缓解该问题。
正文
Authors:Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong
Abstract:The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in Key tensors, its efficacy diminishes under extreme compression. In this work, we revisit the inherent limitations of the per-channel quantization paradigm from both empirical and theoretical perspectives. Our analysis identifies Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity. We demonstrate that TNI systematically amplifies errors when shared quantization parameters are required to span token groups exhibiting substantial norm disparities. Instead of relying on intricate quantization pipelines (e.g., TurboQuant), we propose OScaR (Omni-Scaled Canalized Rotation), an accurate and lightweight KV cache compression framework for X-LLMs (i.e., text-only, multi-modal, and omni-modal LLMs). Advancing the per-channel paradigm, OScaR employs Canalized Rotation followed by Omni-Token Scaling to mitigate TNI-induced sequence-dimensional variance both effectively and efficiently, further supported by our optimized system design and CUDA kernels. Extensive evaluations across X-LLMs show that OScaR consistently outperforms existing methods and achieves near-lossless performance under INT2 quantization, establishing it as a robust, low-complexity, and universal framework that defines a new Pareto front. Compared with the BF16 FlashDecoding-v2 baseline, our OScaR implementation achieves a notable up to 3.0x speedup in decoding, reduces memory footprint by 5.3x, and increases throughput by 4.1x. The code for OScaR is publicly available at this https URL.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.19660 [cs.LG] |
| (or arXiv:2605.19660v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19660 arXiv-issued DOI via DataCite |
Submission history
From: Zunhai Su [view email]
[v1]
Tue, 19 May 2026 10:53:03 UTC (13,912 KB)
[v2]
Sat, 10 Oct 2026 04:19:44 UTC (13,937 KB)
[v3]
Wed, 14 Oct 2026 05:26:57 UTC (13,937 KB)
来源：arXiv：cs.LG · arxiv.org