HuggingFace Daily Papers·· 1 天前
S3AC：面向 LLM 推理的稳定、选择性、可扩展 Actor-Critic 方法
S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning
AI 导读
S3AC 提出一套实用的 critic 训练方案，为 LLM 长链式推理的强化学习提供 token 级优势和 prompt 级基线，并据此提出 Difficulty-Aware Prompt Sampling（DAPS），在生成前筛选提示词以减少 rollout 浪费。
正文 · 原文
Abstract:Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.
| Comments: | 17 pages, 14 figures |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35518 [cs.LG] |
| (or arXiv:2610.35518v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35518 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuo Shen [view email]
[v1]
Mon, 26 Oct 2026 14:38:53 UTC (517 KB)
来源：HuggingFace Daily Papers · arxiv.org