arXiv：cs.AI· Jinhyeok Kim, Hye-Young Jung·· 5 小时前AI 评分34
惩罚框架下的无有效选项 MCQA：分析 LLM 在无效选项下的弃答行为
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
AI 导读
研究提出"惩罚框架下的无有效选项 MCQA"，用 MMLU-Pro 数学子集移除正确选项，允许模型选择剩余选项或输出 ABSTAIN，并对无效强制选择施加惩罚。实验显示，即便有明确的无有效选项指令和惩罚评分，模型仍在部分原本答对的题目上给出无效强制选择，说明高 MCQA 准确率无法完全保证弃答可靠性。
正文
Abstract:Multiple-choice question answering (MCQA) is commonly used to evaluate large language models under the assumption that one of the provided options is correct, typically using answer-selection accuracy. However, in real deployments, users or retrieval systems may provide invalid option sets in which none of the listed choices is correct, and selecting one of them may incur downstream cost. We study this setting as penalty-framed no-valid-option MCQA. Using the mathematics subset of MMLU-Pro, we remove the labeled correct option, allow models to either choose a remaining option or output ABSTAIN, and penalize invalid forced-choice responses. We further introduce correct-conditioned analysis, evaluating abstention only on instances that the model originally answered correctly. Experiments show that high MCQA accuracy does not fully guarantee abstention reliability: even under explicit no-valid-option-aware instructions and penalty-based scoring, models still produce invalid forced-choice responses for a subset of originally correct instances. These results show that penalty-framed no-valid-option MCQA reveals an aspect of model reliability not captured by standard answer-selection accuracy.
| Comments: | Accepted to AACL-IJCNLP 2026 Main Conference (Short Paper) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.08153 [cs.CL] |
| (or arXiv:2610.08153v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08153 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hye-Young Jung [view email]
[v1]
Tue, 6 Oct 2026 11:04:20 UTC (225 KB)
来源：arXiv：cs.AI · arxiv.org