arXiv：cs.CL· Zhaoyan Jia, Chengwei Zhou, Gourav Datta, Massoud Pedram·· 16 小时前
ForkGRPO：面向扩散语言模型的高效探索与细粒度信用分配
ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models
AI 导读
ForkGRPO 是一个面向扩散语言模型（dLLM）的强化学习框架，通过从高熵中间状态采样多条续写并结合段级信用分配来提升推理能力。在四个基准中的三个上，它的性能优于同类架构保持型 RL 方法，比 JustGRPO 最高提升 1.2 个百分点，生成 token 最多减少 27.8%。ForkGRPO-Fast 进一步实现 2.49 倍训练墙钟加速，精度损失可忽略。
正文
Abstract:Diffusion large language models (dLLMs) have advanced rapidly, yet strengthening their reasoning through reinforcement learning (RL) remains an open challenge. Any-order generation can suffer from entropy degradation, which narrows the reasoning paths available for exploration. Recent work mitigates this issue by applying standard GRPO with left-to-right generation. However, independently generating every rollout from the input prompt can leave alternative continuations from intermediate states underexplored, while trajectory-level advantages assign uniform credit to all tokens without distinguishing the contributions of individual reasoning segments. We propose ForkGRPO, an RL framework that couples efficient exploration with fine-grained segment-level credit assignment. It samples multiple continuations from high-entropy intermediate states along a reference trajectory, prioritizing textual boundaries and reusing shared prefixes. The continuations provide complementary learning signals: within-state group-relative advantages guide which continuations to reinforce, while Monte Carlo state-value differences indicate which reasoning segments improve expected outcomes. Among the evaluated architecture-preserving RL methods, ForkGRPO achieves the best performance on three of four benchmarks, while improving over the directly comparable JustGRPO by up to 1.2 percentage points and using up to 27.8% fewer generated tokens. ForkGRPO-Fast further achieves a $2.49\times$ training wall-clock speedup with negligible accuracy loss.
| Comments: | 20 pages, 5 figures. The first two authors contributed equally |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19408 [cs.CL] |
| (or arXiv:2610.19408v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19408 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhaoyan Jia [view email]
[v1]
Wed, 14 Oct 2026 01:56:46 UTC (760 KB)
来源：arXiv：cs.CL · arxiv.org