arXiv：cs.AI· Karl Hanna, Chen Feng·· 23 小时前
LLM 多项选择评测中准确率与顺序敏感性为何背离？GPT 与 Llama 在 MMLU、ARC-Challenge 上的实证研究
A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
AI 导读
研究在 MMLU 和 ARC-Challenge 上测试两个闭源模型与两个开源权重模型，发现准确率与顺序敏感性并不同步变化：两阶段提示词降低了 GPT 在 MMLU 上的 flip rate 和 RStd，准确率却无显著改变；为直接作答加入推理后，Llama 在 MMLU 上的 flip rate 统计上无差异，RStd 却大幅下降。
正文
Abstract:Multiple-choice question (MCQ) benchmarks are widely used to evaluate large language models, yet a single reported accuracy number is not enough to distinguish a model's latent ability from order sensitivity. Debiasing methods tend to be judged on one axis, which implicitly assumes that fixing order sensitivity and improving accuracy go together. We test this assumption with two proprietary models and two open-weight models on MMLU and ARC-Challenge, across four families of strategies. We measure order sensitivity with flip rate, capturing per-question instability, and recall standard deviation (RStd), capturing aggregate positional imbalance. We find accuracy and order sensitivity do not move together consistently: two-stage prompting reduces both flip rate and RStd for GPT on MMLU without a significant change in accuracy. The two sensitivity metrics also diverge from each other. Adding reasoning to direct answering leaves Llama's MMLU flip rate statistically indistinguishable while RStd falls sharply. A $2\times2$ ablation across generation-then-matching, crossing option visibility with matcher type, shows that withholding options mainly harms semantic matching, which achieves zero flip rate by construction, yet loses significantly in accuracy; the effect is smaller when using an LLM matcher. We recommend reporting flip rate and RStd along with accuracy when evaluating MCQ strategies.
| Comments: | Submitted to ARR, aimed at NAACL2027. Code available at this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.11947 [cs.CL] |
| (or arXiv:2608.11947v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.11947 arXiv-issued DOI via DataCite |
Submission history
From: Karl Hanna [view email]
[v1]
Wed, 12 Aug 2026 11:34:45 UTC (77 KB)
[v2]
Sat, 10 Oct 2026 23:56:33 UTC (79 KB)
来源：arXiv：cs.AI · arxiv.org