arXiv：cs.CL· Sangdo Han, Moonsoo Kim·· 11 小时前
提示词优化器是否会破坏未被评分的需求？一项预注册研究
Do Prompt Optimizers Break Requirements They Are Not Scored On? A Preregistered Study
AI 导读
预注册研究在 HotpotQA、SQuAD 2.0 和 IFEval 上测试 GEPA、MIPROv2、TextGrad，按预设标准三者均出现回归；在 HotpotQA 上引用有效率净下降 10.7 至 41.6 分，而 exact match 上升 21 至 34 分。
正文
Abstract:Prompt optimizers rewrite a prompt to raise one target metric, but prompts also state requirements the metric does not score, such as an output format, citations, or abstention. In a preregistered study, we run GEPA, MIPROv2, and TextGrad on HotpotQA, SQuAD 2.0, and IFEval and count per-item violations of unscored requirements with deterministic checks. By the preregistered criterion, all three optimizers produce regressions, substantially on HotpotQA, where net regression in citation validity reaches 10.7 to 41.6 points while exact match rises by 21 to 34 points. A post-hoc decomposition changes this reading. The optimized prompts elicit reasoning before the required JSON object, and under the 200-token output cap used throughout, 13.9% to 47.0% of responses end before any answer appears. Among parseable outputs, net regression is within 3 points of zero. With a 1,024-token cap, extraction failures disappear, content regression stays below 1 point, and exact match rises further. Two simple gates fail the preregistered criterion. The side effect is narrower than hypothesized: optimized prompts are more accurate and break an explicit output contract, and a lenient target metric does not show the cost.
| Comments: | 8 pages, 7 tables. Preregistered study |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13287 [cs.CL] |
| (or arXiv:2610.13287v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13287 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sangdo Han [view email]
[v1]
Fri, 9 Oct 2026 09:38:51 UTC (31 KB)
来源：arXiv：cs.CL · arxiv.org