arXiv：cs.AI（全量分类）· Timoth\'ee Lesort, Alejandra L\'opez de Aberasturi G\'omez, Tristan Karch, Tom Veniat, Philippe Modard, Karl Tuyls, Ludovic Denoyer·· 5 小时前AI 评分41
用国际象棋基准测试大语言模型的提示词优化
Benchmarking Prompt Optimization of Large Language Models With Chess
AI 导读
研究者基于 1118 道 Lichess 谜题构建了一个国际象棋基准，用于研究冻结 LLM 的自动提示词优化（APO），在不更新模型权重的前提下优化提示词。该基准在 8 个目标模型上评测了 6 种 APO 算法，并考察优化提示词能否跨模型迁移及迁移到对局表现。
正文
Abstract:Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55\% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around \$800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts (this https URL).
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.00416 [cs.AI] |
| (or arXiv:2610.00416v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00416 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Timothée Lesort [view email]
[v1]
Wed, 30 Sep 2026 14:25:55 UTC (224 KB)
来源：arXiv：cs.AI（全量分类） · arxiv.org