arXiv：cs.LG· Zhen Li, Shuai Zhang, Yanggan Gu, Yiming Zhang, Yang Yu, Mingfa Feng, Congkai Xie, Shuang Yu, Junjie Lai, Hongxia Yang·· 4 小时前AI 评分34
TRIAGE：面向原生 NVFP4 强化学习的方向感知失配稳定方法
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
AI 导读
TRIAGE 是一种方向感知的稳定方法，用于解决原生 NVFP4 强化学习中学习器与采样器执行失配导致的策略优化不稳定问题。该方法通过段级诊断选择性重新平衡策略梯度更新，并对残余严重失配进行有界修复，同时保留采样器与学习器上的原生 NVFP4 W4A4 前向执行。
正文
Abstract:Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3x higher rollout throughput than BF16.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07043 [cs.LG] |
| (or arXiv:2610.07043v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07043 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhen Li [view email]
[v1]
Mon, 5 Oct 2026 00:53:11 UTC (1,669 KB)
来源：arXiv：cs.LG · arxiv.org