arXiv：cs.LG· Nicolas Hoischen, Petar Bevanda, Max Beier, Stefan Sosnowski, Boris Houska, Sandra Hirche·· 13 小时前
Infinitesimal Operator World Models：离线强化学习的有限样本保证
Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning
AI 导读
研究者提出 Infinitesimal Operator World Models，将强化学习与 Hamilton-Jacobi-Bellman 方程关联，在再生核希尔伯特空间中用受控扩散过程的无穷小生成元表示世界模型。在生成元近似误差足够小时，该方法可证明价值函数的全局收敛，并给出与系统平滑性和稳定性直接绑定的显式有限样本保证。
正文
Abstract:Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG); Systems and Control (eess.SY); Optimization and Control (math.OC) |
| Cite as: | arXiv:2511.10383 [stat.ML] |
| (or arXiv:2511.10383v2 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2511.10383 arXiv-issued DOI via DataCite |
|
| Journal reference: | IEEE Open Journal of Control Systems, 01 October 2026 |
| Related DOI: | https://doi.org/10.1109/OJCSYS.2026.3739678
DOI(s) linking to related resources |
Submission history
From: Nicolas Hoischen [view email]
[v1]
Thu, 13 Nov 2025 14:58:30 UTC (1,427 KB)
[v2]
Fri, 9 Oct 2026 13:08:08 UTC (2,717 KB)
来源：arXiv：cs.LG · arxiv.org