arXiv：cs.LG· Chung Min Kim, Brent Yi, David McAllister, Hongsuk Choi, Himanshu Gaurav Singh, Jinkun Cao, Ken Goldberg, Pieter Abbeel, Carmelo Sferrazza, Angjoo Kanazawa·· 7 小时前AI 评分42
QF3：用过滤 Q 梯度的快速流式强化学习
QF3: Fast Flow RL with Filtered Q-Gradients
AI 导读
QF3 是一种在线离策略强化学习算法，通过流匹配与 critic 动作梯度训练流策略，梯度经流输出的一步预测反向传播，且只作用于接近回放动作的维度。它是首个从零训练人形机器人运动策略并零样本迁移到硬件的离策略流 RL 方法，训练速度比 FPO++ 快 10 倍，还可在 ABC-Sim 和 Robomimic 上微调预训练流式操作策略。
正文
Abstract:Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where the critic and this prediction are reliable, QF3 applies the critic gradient only to action dimensions that stay near the replay action. To our knowledge, QF3 is the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware. Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method. We further apply QF3 to fine-tune pretrained flow-based manipulation policies on both ABC-Sim and Robomimic tasks. These results suggest that QF3 can both learn robot policies from scratch and refine those acquired from demonstrations. Website: this https URL
| Comments: | Project page: this https URL |
| Subjects: | Robotics (cs.RO); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.08789 [cs.RO] |
| (or arXiv:2610.08789v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08789 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chung Min Kim [view email]
[v1]
Tue, 6 Oct 2026 17:59:34 UTC (8,125 KB)
来源：arXiv：cs.LG · arxiv.org