arXiv：cs.LG· Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You·· 11 小时前
DiffusionART：高效、稳定且可扩展的扩散模型强化学习
DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models
AI 导读
DiffusionART 提出“聚合优先奖励倾斜”，将每个 prompt 的奖励聚合为反向过程状态共享的奖励改进方向，从构造上消除 NFT 类目标中隐藏的正反向一致性力，并加入信任域掩码的二次更新实现漂移自适应恢复，无需 EMA 调参。
正文
Abstract:Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.
| Comments: | Project page: this https URL ; Code: this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12572 [cs.LG] |
| (or arXiv:2610.12572v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12572 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhenglin Wan [view email]
[v1]
Thu, 8 Oct 2026 18:01:44 UTC (17,685 KB)
来源：arXiv：cs.LG · arxiv.org