arXiv：cs.LG· Robin Uhrich, Marco Bagatella, N\'uria Armengol Urp\'i, Pavel Kolev, Georg Martius·· 4 小时前
用 Drifting 建模 Occupancy：一步生成实现直接策略提取
Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction
AI 导读
研究提出用一步生成采样器 drifting 对 occupancy（折扣状态访问分布）建模，从而通过 occupancy 模型直接微分完成策略提取，绕过 critic 蒸馏。作者沿 occupancy 多模态结构、生成模型类别、策略提取方法三条轴拆解该配方，逐一实验验证并提出更简化的替代方案，在标准 benchmark 上取得性能提升。
正文
Abstract:The growing interest in behavior foundation models, combined with recent advancements in generative modeling, has led to an appealing recipe for reinforcement learning from unlabeled data: pre-train a generative model to model occupancies, i.e., discounted state visitations, on reward-free trajectories, such that a policy can be easily extracted once a small reward-labeled dataset becomes available. We decompose this recipe along three axes: (i) the degree to which the multi-modal structure of occupancies must be captured, (ii) the class of generative model, and (iii) the method used for policy extraction. Along each axis, we consider existing choices, propose simpler alternatives, and validate each decision experimentally. This analysis culminates in a novel, streamlined method that leverages drifting, a modern one-step generative sampler, to model the occupancy. We show that this enables policy extraction via direct differentiation through the occupancy model, and bypasses the need for critic distillation. Overall, we observe that this nimble setup leads to improvements in performance on standard benchmarks.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19883 [cs.LG] |
| (or arXiv:2610.19883v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19883 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Robin Uhrich [view email]
[v1]
Wed, 14 Oct 2026 09:37:20 UTC (742 KB)
来源：arXiv：cs.LG · arxiv.org