arXiv：cs.AI· Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang·· 17 小时前
KV4-RL：用 INT4 KV Cache 量化加速 RLVR 异步 rollout
KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
AI 导读
KV4-RL 将免校准 INT4 KV 量化与对 attention sinks 及近期上下文的 INT8 选择性保护结合，缓解 INT4 rollout 的学习退化。
正文
Abstract:Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this \emph{KV concentration} and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose \emph{KV4-RL}, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout, reaching $1.97\times$ rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.
| Comments: | 32 pages, 15 figures, 6 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14448 [cs.LG] |
| (or arXiv:2610.14448v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14448 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sihwan Park [view email]
[v1]
Sat, 10 Oct 2026 01:24:03 UTC (866 KB)
来源：arXiv：cs.AI · arxiv.org