arXiv：cs.AI· Xiang Li, Yuheng Zhang, Nan Jiang·· 13 小时前
超越逐状态镜像下降：基于参数化策略的离线策略优化
Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
AI 导读
该研究将离线强化学习的理论保证扩展到大规模或连续动作空间上的参数化策略类，突破了PSPI等算法仅适用于有限小动作空间且依赖逐状态镜像下降、需从critic隐式导出actor的限制。作者识别出上下文耦合是镜像下降扩展到参数化策略的核心难点，并揭示其与自然策略梯度的联系，由此提出新分析与算法洞见，包括离线RL与模仿学习之间的意外统一。
正文
Abstract:We investigate the theoretical aspects of offline reinforcement learning (RL) under general function approximation. While prior works (e.g., Xie et al., 2021) have established the theoretical foundations of learning a good policy from offline data via pessimism, existing algorithms that are computationally tractable (often in an oracle-efficient sense), such as PSPI, only apply to finite and small action spaces. Moreover, these algorithms rely on state-wise mirror descent and require actors to be implicitly induced from the critic functions, failing to accommodate standalone policy parameterization which is ubiquitous in practice. In this work, we address these limitations and extend the theoretical guarantees to parameterized policy classes over large or continuous action spaces. When extending mirror descent to parameterized policies, we identify contextual coupling as the core difficulty, and show how connecting mirror descent to natural policy gradient leads to novel analyses, guarantees, and algorithmic insights, including a surprising unification between offline RL and imitation learning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.23811 [cs.LG] |
| (or arXiv:2602.23811v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.23811 arXiv-issued DOI via DataCite |
Submission history
From: Xiang Li [view email]
[v1]
Fri, 27 Feb 2026 08:51:57 UTC (100 KB)
[v2]
Tue, 3 Mar 2026 02:13:05 UTC (100 KB)
[v3]
Wed, 25 Mar 2026 05:18:26 UTC (96 KB)
[v4]
Thu, 7 May 2026 23:18:19 UTC (80 KB)
[v5]
Fri, 9 Oct 2026 16:28:54 UTC (100 KB)
来源：arXiv：cs.AI · arxiv.org