arXiv：cs.LG· Kintan Saha, Ahana Deb, Anders Jonsson, Debabrota Basu·· 4 小时前
平均奖励 Q-learning 与 options 的非渐近收敛性研究
Non-asymptotic Convergence of Average-reward Q-learning with Options
AI 导读
研究首次为平均奖励框架下的 option Q-learning 算法给出均方意义下 Õ(1/iterations) 的收敛保证，覆盖 inter-option 与 intra-option 两种设定。作者证明两类 option Bellman 算子满足多步 span 收缩性质，并提出对应的同步估计量，由此得到两种 option Q-learning 变体，同时将随机逼近框架扩展以证明其收敛速率。
正文
Abstract:We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options. While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL. We establish the first $\tilde{O}(1/\text{iterations})$ convergence guarantees, in the mean-square sense, for average-reward option Q-learning algorithms in both the inter- and intra-option settings. First, we prove that both inter- and intra-option Bellman operators satisfy multi-step span contractions under unichain assumptions. Then, we propose synchronous estimators of these multi-step option Bellman operators leading to two variants of option Q-learning. Finally, we extend the stochastic approximation framework to prove convergence rates of these two algorithms under the minimal assumptions used in the literature.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20347 [cs.LG] |
| (or arXiv:2610.20347v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20347 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kintan Saha [view email]
[v1]
Wed, 14 Oct 2026 15:26:23 UTC (2,429 KB)
来源：arXiv：cs.LG · arxiv.org