arXiv：cs.AI· Ning Liu, Chuanneng Sun, Kristina Klinkner, Shervin Malmasi·· 13 小时前
GraphDPO：用偏好图替代成对比较来对齐语言模型
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
AI 导读
研究者提出 GraphDPO，将 DPO 从成对偏好比较推广到由 rollout 排名诱导的有向无环偏好图，用图结构 Plackett-Luce 目标聚合支配邻域监督并传播传递性，标准 DPO 是其特例。
正文
Abstract:Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.08037 [cs.LG] |
| (or arXiv:2605.08037v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08037 arXiv-issued DOI via DataCite |
Submission history
From: Ning Liu [view email]
[v1]
Fri, 8 May 2026 17:26:14 UTC (167 KB)
[v2]
Thu, 8 Oct 2026 18:14:25 UTC (183 KB)
来源：arXiv：cs.AI · arxiv.org