arXiv：cs.LG（机器学习，全量分类）· Mintae Kim, Koushil Sreenath·· 5 小时前AI 评分30
IDI：面向基于模型的离线强化学习的分布内想象
In-Distribution Imagination for Model-Based Offline Reinforcement Learning
AI 导读
研究者提出分布内想象（IDI），一种在学习到的表示空间中估计轨迹支撑度、并自适应截断偏离离线轨迹流形的 rollout 控制框架。实验显示，轨迹支撑度对 rollout 失败的预测显著优于 transition 级不确定性，结合轨迹正则化 RL 后，IDI 在有限数据设置下持续提升性能。
正文
Abstract:Model-based offline reinforcement learning (MBORL) improves sample efficiency through model-generated trajectories. However, accumulative model error can drive imagined trajectories outside the offline data distribution, leading to unrealistic synthetic data and unstable policy optimization. Many existing methods primarily control rollouts using transition-level uncertainty. We propose \emph{in-distribution imagination} (IDI), a rollout control framework that estimates trajectory support in a learned representation space and adaptively truncates rollouts that leave the offline trajectory manifold. Combined with trajectory-regularized RL, an extension of entropy-regularized RL, IDI consistently improves performance in limited-data settings. Experiments show that trajectory support predicts rollout failure substantially better than transition-level uncertainty, highlighting the importance of trajectory-level rollout control in MBORL.
| Comments: | 11 pages, 3 figures, RLC 2026 MBRL Workshop |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.38673 [cs.LG] |
| (or arXiv:2609.38673v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38673 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mintae Kim [view email]
[v1]
Tue, 29 Sep 2026 23:57:02 UTC (218 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org