arXiv：cs.AI· Riccardo Formenti, Aditya Agrawal, Vikas Garg·· 4 小时前
你的偏好模型可信吗？用 Stackelberg 博弈做对齐
Can You Trust Your Preference Models? Stackelberg Games for Alignment
AI 导读
针对语言模型对齐中评估器错误可被优化利用的问题，研究者用偏好模型集成间的分歧作为预测误差的代理，将寻找可被反转偏好的比较样本建模为 Leader 与 Follower 之间的零和 Stackelberg 博弈，并提出两时间尺度算法 SMIDA，在表格设置下具备线性 last-iterate 收敛，另有神经策略梯度版本。
正文
Abstract:Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.
| Subjects: | Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT) |
| Cite as: | arXiv:2610.15008 [cs.AI] |
| (or arXiv:2610.15008v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15008 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Riccardo Formenti [view email]
[v1]
Sat, 10 Oct 2026 15:41:36 UTC (369 KB)
来源：arXiv：cs.AI · arxiv.org