arXiv：cs.LG· Adrien Bolland, Gaspard Lambrechts, Damien Ernst·· 4 小时前
基于未来状态-动作访问测度的最大熵探索
Maximum-Entropy Exploration with Future State-Action Visitation Measures
AI 导读
该研究提出以未来时间步访问的状态-动作特征折扣分布熵作为内在奖励，证明了其期望和是初始状态轨迹中该特征分布熵的下界，且该分布是某收缩算子的不动点，可离策略估计。实验显示新目标常提升单条轨迹内的特征访问多样性，但会略微降低跨轨迹的期望访问，同时加快纯探索智能体的收敛速度；在所用基准上大多数方法的控制性能相近。
正文
Abstract:Maximum entropy reinforcement learning motivates agents to explore states and actions to maximize the entropy of some distribution, typically by providing additional intrinsic rewards proportional to that entropy function. In this paper, we study intrinsic rewards proportional to the entropy of the discounted distribution of state-action features visited during future time steps. This approach is motivated by two results. First, we show that the expected sum of these intrinsic rewards is a lower bound on the entropy of the discounted distribution of state-action features visited in trajectories starting from the initial states, which we relate to an alternative maximum entropy objective. Second, we show that the distribution used in the intrinsic reward definition is the fixed point of a contraction operator and can therefore be estimated off-policy. Experiments highlight that the new objective often leads to improved visitation of features within individual trajectories, in exchange for slightly reduced visitation of features in expectation over different trajectories, as suggested by the lower bound. It also leads to improved convergence speed for learning exploration-only agents. Control performance remains similar across most methods on the considered benchmarks.
| Comments: | arXiv admin note: substantial text overlap with arXiv:2412.06655 |
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2603.18965 [cs.LG] |
| (or arXiv:2603.18965v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.18965 arXiv-issued DOI via DataCite |
Submission history
From: Adrien Bolland [view email]
[v1]
Thu, 19 Mar 2026 14:30:08 UTC (17,231 KB)
[v2]
Tue, 13 Oct 2026 18:32:32 UTC (17,233 KB)
来源：arXiv：cs.LG · arxiv.org