PASS：为 LLM 推理模型的过程监督强化学习构建优势信号

View PDF HTML (experimental)

Abstract:Dense process supervision supplies local feedback but leaves open how that feedback should become a policy advantage. We introduce PASS (Process Advantage Signal Shaping), an advantage-construction pipeline for group-relative policy optimization (GRPO). PASS combines separate reward-channel normalization and fusion, signal-defined chunks for credit accumulation, and normalization by the remaining chunk count. This design connects the scale of local feedback to the resolution and horizon over which it contributes to a policy update. We evaluate PASS using learned process rewards for mathematical reasoning and teacher-derived token signals for multi-hop question answering. On seven mathematics benchmarks, PASS improves average pass@1 by $4.62$ points over outcome-only GRPO while reducing average response length by approximately $2\%$. On multi-hop QA with mean-centered normalization, PASS exceeds outcome-only training by $0.87$ and $0.63$ points for two distillation signals, with longer responses. Across both normalization schemes, PASS achieves the highest mean accuracy among the tested process-signal configurations and remains closest to outcome-only under absolute-maximum normalization. These findings show that channel handling, credit resolution, and accumulation should be considered jointly when incorporating dense supervision into policy optimization.

Submission history

From: Chao Wang [view email]
[v1] Sun, 28 Jun 2026 09:36:43 UTC (253 KB)
[v2] Sat, 10 Oct 2026 11:10:57 UTC (267 KB)