arXiv：cs.LG（机器学习，全量分类）· Yaniv Oren, Viliam Vadocz, Wiktor Zabka, Thomas Evers, Jan Robine, Wendelin B\"ohmer, Matthijs T. J. Spaan, Martha White, Hendrik Baier, Fenghui Yu·· 1 天前AI 评分37
迈向最优策略改进：近似评估下的最优贪心化算子
Towards Optimal Policy Improvement
AI 导读
研究从第一性原理定义最优策略改进，证明限定状态集上的最优改进等价于求解一个诱导 MDP，并将显式或隐式模型的规划刻画为通往最优策略改进的路径。针对近似评估这一核心约束，作者将贪心化建模为不确定性下的概率决策，推导出一个对该目标最优的新算子。
正文
Abstract:Practical Reinforcement Learning (RL) algorithms learn to solve Markov Decision Processes (MDPs) through iterative policy improvement in the presence of approximate evaluation. We study policy improvement from first principles, defining optimal policy improvement as producing the best policy attainable in a single update under specified constraints. We show that optimal improvement restricted to a set of states is equivalent to solving an induced MDP, characterizing planning with an explicit or implicit model as a path towards optimal policy improvement. Because practical methods commonly solve such induced problems through iterative improvement in the form of greedification, we take steps towards optimal greedification under the central practical constraint of approximate evaluation. We formulate greedification under this constraint as probabilistic decision-making under uncertainty and derive a novel operator that is optimal with respect to the resulting objective. Empirically, the operator and its practical gradient-based approximations improve aggregate performance across GumbelAlphaZero, SAC, ReBRAC and Generalized Policy Iteration, in experiments spanning discrete and continuous actions, model-based and model-free, online and offline RL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.01566 [cs.LG] |
| (or arXiv:2610.01566v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.01566 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yaniv Oren [view email]
[v1]
Thu, 1 Oct 2026 12:29:34 UTC (546 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org