arXiv：cs.CL· Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw·· 10 小时前
HERO：用执行证据做测试时精修与提示词优化的程序生成方法
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
AI 导读
HERO 结合离线与测试时 harness 优化来提升 LLM 的程序生成能力：第一阶段从执行验证的成功与失败中学习可复用提示词更新，并在采纳前验证候选修订；测试时把生成的代码当作实例级 harness，用执行反馈继续精修。在 SQL、Cypher、Python 四个基准上，HERO 取得最佳离线表现，pass@1 较初始提示词提升 7.6 分，测试时精修再带来 2.0 分提升并改善执行效率。
正文
Abstract:Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \textsc{HERO} (\textbf{H}arnessing \textbf{E}vidence for \textbf{R}efinement and \textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \method{} achieves the best offline performance, improving pass@1 by \(7.6\) points over the initial prompt, while test-time refinement adds another \(2.0\) points and improves execution efficiency.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19184 [cs.CL] |
| (or arXiv:2610.19184v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19184 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: MingYu Lu [view email]
[v1]
Tue, 13 Oct 2026 20:49:08 UTC (636 KB)
来源：arXiv：cs.CL · arxiv.org