arXiv：cs.CL· Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou·· 16 小时前
DERL：双演化强化学习实现自动化奖励设计
Dual-Evolving Reinforcement Learning for Automated Reward Design
AI 导读
研究者提出 Dual-Evolving Reinforcement Learning（DERL），一种双训练框架，让外层提议器通过策略梯度从内层策略的验证表现中学习，从而自主发现奖励结构。
正文
Abstract:Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.
| Comments: | Work in Progress. We release our code and model at this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.13399 [cs.AI] |
| (or arXiv:2512.13399v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2512.13399 arXiv-issued DOI via DataCite |
Submission history
From: Sitao Cheng [view email]
[v1]
Mon, 15 Dec 2025 14:50:08 UTC (583 KB)
[v2]
Wed, 13 May 2026 04:43:51 UTC (646 KB)
[v3]
Tue, 13 Oct 2026 21:12:48 UTC (1,275 KB)
来源：arXiv：cs.CL · arxiv.org