arXiv：cs.LG· Zhengyan Wan, Yidong Ouyang, Panwen Hu, Qiang Sun·· 2 天前AI 评分35
dFlowGRPO：面向离散流模型的速率感知策略优化
dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models
AI 导读
研究者提出 dFlowGRPO，一个面向离散流模型的统一强化学习框架，支持多种概率路径与非掩码源分布，通过推导完整轨迹概率并将去噪过程建模为马尔可夫决策过程，同时利用条件转移速率和后验模型信息。将该方法应用于多模态离散流模型 FUDOKI 后，其在文生图任务上优于现有面向 dLLM 的 GRPO 类方法，性能与用 Flow-GRPO 训练的连续流模型相当，并在理解任务上表现强劲。
正文
Abstract:Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of a mixture path and a masked source distribution. While several recent works have explored reinforcement learning for dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate the denoising process as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using Flow-GRPO, while also demonstrating strong capabilities on understanding tasks.
| Subjects: | Machine Learning (cs.LG); Applications (stat.AP) |
| Cite as: | arXiv:2605.09291 [cs.LG] |
| (or arXiv:2605.09291v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09291 arXiv-issued DOI via DataCite |
Submission history
From: Zhengyan Wan [view email]
[v1]
Sun, 10 May 2026 03:36:49 UTC (8,024 KB)
[v2]
Thu, 1 Oct 2026 03:41:55 UTC (7,925 KB)
来源：arXiv：cs.LG · arxiv.org