arXiv：cs.AI· Yaomin Wang, Jianting Pan, Ran Tian, Xiaoyang Li, Yu Zhang, Hengle Qin, Tianshu YU·· 20 小时前
AdaGamma：面向强化学习时间自适应的状态依赖折扣
AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning
AI 导读
AdaGamma 是一个在深度 actor-critic 强化学习中实现状态依赖折扣的框架，通过学习有界折扣函数 γ_φ(s)，并用回报一致性目标训练，避免 TD-collapse。该方法可集成进 SAC 和 PPO，在连续控制基准上取得提升，并在 JD Logistics 平台为期四周的在线 A/B 测试中获得统计显著收益。
正文
Abstract:The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. State-dependent discounting is a natural way to adapt temporal credit assignment to local state structure, but naively learning a discount function inside deep actor--critic methods creates a self-referential target: the learned discount scales the same bootstrap term against which it is trained. Consequently, TD-error-based objectives can admit a degenerate shortcut, reducing target dependence on downstream value estimates instead of learning meaningful temporal structure. We propose \textbf{AdaGamma}, a practical framework for state-dependent discounting in deep actor--critic reinforcement learning. AdaGamma learns a bounded discount function \(\gamma_\phi(s)\) and trains it using a return-consistency objective that matches a one-step adaptive bootstrap to a multi-step return computed with a lagged, stop-gradient reference discount. This removes the direct target manipulation path and encourages the discount to encode state-dependent value propagation. We analyze the Bellman operator induced by a fixed state-dependent discount in finite tabular MDPs, proving contraction, exact soft policy iteration, and Lipschitz sensitivity under standard boundedness assumptions. Empirically, AdaGamma integrates into SAC and PPO, avoids TD-collapse, improves continuous-control benchmarks, and achieves statistically significant gains in a four-week online A/B test on the JD Logistics platform.
| Comments: | 29 pages, 6 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.06149 [cs.LG] |
| (or arXiv:2605.06149v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.06149 arXiv-issued DOI via DataCite |
Submission history
From: Yaomin Wang [view email]
[v1]
Thu, 7 May 2026 12:42:46 UTC (11,758 KB)
[v2]
Sun, 11 Oct 2026 03:53:21 UTC (11,840 KB)
来源：arXiv：cs.AI · arxiv.org