arXiv：cs.LG· Denis Blessing, Onur Celik, Gerhard Neumann·· 11 小时前
基于不动点流匹配的强化学习
Reinforcement Learning via Fixed-point Flow Matching
AI 导读
研究提出不动点流匹配（FFM），一种无需训练数据即可在强化学习中训练和微调流策略的方法，其利用目标流恒等式将 actor 更新转化为不动点迭代，以 critic 的动作梯度作为回归目标。实验表明，FFM 能在离线到在线场景中有效微调预训练流策略，并支持在线 RL 中从零高效训练流策略。
正文
Abstract:In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.
| Subjects: | Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2610.12848 [cs.LG] |
| (or arXiv:2610.12848v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12848 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Denis Blessing [view email]
[v1]
Fri, 9 Oct 2026 01:15:10 UTC (1,951 KB)
来源：arXiv：cs.LG · arxiv.org