arXiv：cs.CL· Jinjin He, Xinsong Feng, Lucheng Fu, Sinan Wang, Bo Zhu, Jiayun Wang·· 11 小时前
重新审视扩散与解码的角色分工：面向并行语言生成的 Predictive-State Flows
Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation
AI 导读
针对连续扩散语言模型（DLM）中扩散先验需在连续空间捕捉细粒度离散 token 选择的问题，研究者提出 Predictive-State Flows（PSF），将序列分块并为每块关联一个仅由前文计算出的预测状态，由 rectified-flow 先验生成预测状态序列，再由块因果解码器并行生成所有块、仅在块内自回归。
正文
Abstract:In continuous diffusion language models (DLMs), latents generated by the diffusion prior are mapped almost deterministically to tokens, either by a decoder or through a token embedding matrix. Unlike in image generation, where visual details can vary continuously, the diffusion prior must capture fine-grained discrete token choices in continuous space, which can make the joint latent distribution difficult to model. We argue that generative roles should instead be divided: the diffusion prior should generate high-level predictive information, while the decoder should resolve the remaining token-level choices. Based on this view, we introduce Predictive-State Flows (PSF), which partitions a sequence into blocks and associates each block with a predictive state computed solely from the preceding context. A rectified-flow prior generates the sequence of predictive states, after which a block-causal decoder generates all blocks in parallel, autoregressively only within each block. This design reduces diffusion from L token-level states to K << L predictive states and reduces token-level sequential depth to L/K. On unconditional OpenWebText generation, a 197M-parameter model with K=8 achieves a generative perplexity of 22.8 at entropy 5.16 using 0.31 TFLOPs per 1024-token document, versus 24.0 and 7.2 TFLOPs for ELF-B with 32 sampling steps. The corresponding measured operating points yield MAUVE@512 of 0.93 and 0.73, respectively. Controlled comparisons further show that content states, which additionally encode their own blocks, yield 8-16% higher generative perplexity than predictive states, and that replacing sampled predictive-state trajectories with a shared mean lowers MAUVE@512 by 0.11. These results suggest that latent design for continuous DLMs should consider not only how compactly text is represented, but also how generative roles are divided between the prior and the decoder.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12898 [cs.CL] |
| (or arXiv:2610.12898v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12898 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinjin He [view email]
[v1]
Fri, 9 Oct 2026 02:43:30 UTC (221 KB)
来源：arXiv：cs.CL · arxiv.org