arXiv：cs.LG· Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao, Wenhuan Sun, Xiaojiao Yu, Suryabhan Singh Hada, Libin N. George, Uma Kona, Richard Williamson, Linsey Pang, Prakhar Mehrotra·· 4 小时前AI 评分37
CM-DPO：面向 LLM 规划的约束边际直接偏好优化
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning
AI 导读
CM-DPO 将 DPO 的二值偏好信号替换为由确定性符号验证器生成、并按违规严重程度加权的连续边际，通过字典序目标分离硬约束与软约束。配套 SynPlan-R 框架用程序化约束画像（DCCG）和推理教师最小编辑蒸馏（RT-MED）生成低偏差偏好数据。
正文
Authors:Rabimba Karanjai, Qun Gu, Hemanth Hegadehalli Madhavarao, Wenhuan Sun, Xiaojiao Yu, Suryabhan Singh Hada, Libin N. George, Uma Kona, Richard Williamson, Linsey Pang, Prakhar Mehrotra
Abstract:Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity. Hard and soft constraints are separated through a lexicographic objective, ensuring hard constraints are never traded off against preferences. To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R. On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.09219 [cs.AI] |
| (or arXiv:2610.09219v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09219 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rabimba Karanjai [view email]
[v1]
Tue, 6 Oct 2026 23:30:17 UTC (404 KB)
来源：arXiv：cs.LG · arxiv.org