arXiv：cs.AI· Hua Lu, Pengcheng Wang, Han Li, Zichen Tian, Yanyu Chen, Huang He, Bingjin Chen·· 22 小时前
SWEPatchTest：超越复现的软件测试评估基准
SWEPatchTest: Evaluating Software Tests Beyond Reproduction
AI 导读
SWEPatchTest 是一个测试生成基准，包含来自 SWE-bench Pro、SWE-bench Multilingual 和 DeepSWE 的 619 个任务及 82,579 个候选补丁，通过联合衡量问题复现、正确补丁接受与错误补丁拒绝来评估测试质量。
正文
Abstract:Coding agents use generated tests to decide whether a proposed patch satisfies an issue's requirements. A suite that fails on the buggy program and passes after applying the reference patch can still accept an incorrect patch or reject a correct patch. We introduce SWEPATCHTEST, a test-generation benchmark with 619 tasks from SWE-bench Pro, SWE-bench Multilingual, and DeepSWE and 82,579 candidate patches collected from coding-agent trajectories. By replaying each suite on the buggy and patched codebases, \benchmark jointly measures issue reproduction, acceptance of correct patches, and rejection of incorrect patches. Joint Validation Rate (JVR), which requires all three criteria on the same task, exposes failures hidden by single-reference evaluation: the best-performing configuration achieves only 9.21% JVR despite reproducing the reference transition on 59.29% of tasks. We further use validation scores to select and scale demonstrations for supervised fine-tuning (SFT). The resulting training recipe approximately triples JVR and improves downstream repair across three SWE-bench benchmarks. Patch discrimination thus provides both a diagnostic measure of test quality and an execution-based signal for curating agent training data.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18004 [cs.SE] |
| (or arXiv:2610.18004v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18004 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zichen Tian [view email]
[v1]
Tue, 13 Oct 2026 09:04:57 UTC (678 KB)
来源：arXiv：cs.AI · arxiv.org