arXiv：cs.LG· Liam Davis, Romain Emil-Popescu, Suman Jana·· 2 小时前
LLM 是否真的回答了问题？一种用于越狱评估的语义重建游戏
Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation
AI 导读
针对 LLM 越狱攻击的 ASR 评估可能将拒绝或仅话题相关的回答误判为攻击成功，仅更换评判 LLM 就能让报告的 ASR 相差最多 82 个百分点。为此研究者提出无需训练的语义重建游戏评估器：仅根据回答重建其实际回应的请求，排除重复或拒绝的内容，再与相似诱饵请求比对，两者都不匹配时可选择弃权。在三种现成 LLM 评判模型上，该方法对人类标签的平均误差最低，并降低了报告 ASR 的波动。
正文
Abstract:Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses that fulfill prohibited requests. Existing evaluators can mistake refusals or topically related responses for successful attacks. We show that changing only the judge LLM can alter reported ASR by up to $82$ percentage points. We introduce the semantic reconstruction game, a training-free evaluator that decomposes compliance judgments into constrained semantic comparisons. Given only a response, an LLM reconstructs the request answered by its substantive content, excluding information merely repeated or refused. The game compares this reconstruction against similar decoy requests to determine whether it matches the original request, allowing abstention when neither candidate matches. Across three off-the-shelf LLM judges, the game achieves the lowest mean error against human labels among evaluated methods and reduces variation in reported ASR. It prioritizes avoiding false positives, even at the cost of missing some successful jailbreaks.
| Subjects: | Machine Learning (cs.LG); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.19306 [cs.LG] |
| (or arXiv:2610.19306v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19306 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Liam Davis [view email]
[v1]
Tue, 13 Oct 2026 23:32:17 UTC (44 KB)
来源：arXiv：cs.LG · arxiv.org