arXiv：cs.LG· Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lianrui Mu, Lei Wang, Chang Liu, Siming Fu·· 13 小时前
FlowAWR：基于优势加权校正的在线自适应流模型强化学习
FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification
AI 导读
FlowAWR 提出一种将连续生成策略优化转化为对理论最优速度场监督回归的新范式，无需 SDE 采样器和 CFG 即可完成优化与生成。在 SD3.5-Medium 上，其对齐性能优于 DiffusionNFT，收敛速度提升 2 至 5 倍，1.2k 步达到 24.12 PickScore，而 DiffusionNFT 需 2.0k 步才达 23.82。
正文
Abstract:Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG).
While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality.
We propose \textit{Flow Advantage-Weighted Rectification} (\textbf{FlowAWR}), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field.
Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation.
In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2$\times$ to 5$\times$ convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in $>$4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance. Code at \href{this https URL}{FlowAWR}.
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2606.30376 [cs.LG] |
| (or arXiv:2606.30376v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.30376 arXiv-issued DOI via DataCite |
Submission history
From: Siming Fu [view email]
[v1]
Mon, 29 Jun 2026 14:37:36 UTC (9,190 KB)
[v2]
Fri, 9 Oct 2026 08:35:47 UTC (15,658 KB)
来源：arXiv：cs.LG · arxiv.org