arXiv：cs.AI· Jianguo Huang, Yao Long Teng, Yanchen Deng, Bo An·· 6 小时前
System One 模型选项顺序偏差：Runoff 让选择题答案与选项排列无关
Consider, Then Choose: Order-Invariant Decisions in System One Models
AI 导读
针对 System One 模型在选择题中答案随选项顺序变化的问题，研究者提出无需训练的 Runoff：先单独给每个选项打分保留少量候选，再让候选在所有位置上两两比较，两次前向传播即可完成，答案按构造与选项顺序无关。
正文
Abstract:System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose $\textbf{Runoff}$, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.18233 [cs.AI] |
| (or arXiv:2610.18233v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18233 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jianguo Huang [view email]
[v1]
Tue, 13 Oct 2026 12:02:46 UTC (192 KB)
来源：arXiv：cs.AI · arxiv.org