arXiv：cs.LG· Xin Wang, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, Jian Luan·· 3 小时前AI 评分37
AdaStep：面向智能体强化学习的自适应步级信用加权
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
AI 导读
AdaStep 是一种自适应步级信用加权方法，通过将权重求解建模为潜在步级优势的均方误差估计问题，在显式条件采样假设下推导出最优的逐状态收缩系数，从而控制组内局部优势对轨迹级信号的修正强度。该方法仅需轻量标量计算，无需 critic、额外 rollout 或额外模型推理。在 ALFWorld、WebShop 和 ScienceWorld 上，三种模型骨干的实验均以低计算成本稳定超越基线。
正文
Abstract:Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose AdaStep, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a signal-to-total-variance interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show consistent improvements over baselines at low computational cost.
| Comments: | 21 pages, 3 figures |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.03223 [cs.LG] |
| (or arXiv:2610.03223v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03223 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xin Wang [view email]
[v1]
Fri, 2 Oct 2026 12:38:53 UTC (384 KB)
来源：arXiv：cs.LG · arxiv.org