arXiv：cs.AI· Jingjing (May), Liu, Dishu Yang, Hongbo Qin, Hansong Zhang·· 4 小时前
测试通过却语义缺失：基于规划的 DataFrame 代码生成中的解释锁定现象
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
AI 导读
研究在 TPC Benchmark DS 的 30 个 DataFrame 任务上发现"解释锁定"现象：测试通过数从 22/30 升至 29/30 和 30/30，但没有任何生成解匹配存储的参考输出，77 次通过测试的运行仍与参考结果不符。
正文
Abstract:Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, comparing Frozen One-Pass, Test-Guided Revision, and PlanAudit-Guided Revision. Test-pass counts rose from 22/30 to 29/30 and 30/30, yet no generated solution matched its stored reference output. Evaluator controls returned the expected outcome in 57/57 cases. Same-engine SQL re-execution reproduced 28/30 references; all 77 test-passing runs on these tasks still failed reference agreement. PlanAudit-Guided Revision improved code scores on 3/30 tasks and test coverage on 11/30, but the median paired change was zero for both measures. Failure analysis revealed tests that omitted requirements, adopted candidate-defined schemas, used nondiscriminating fixtures, or repeated the implementation's incorrect logic. Fresh generation on ten targeted tasks produced 0/30 reference matches and 25/25 false passes among test-passing runs. These findings are diagnostic rather than causal: the pipelines differ in feedback, grounding, call budgets, and editing permissions, and the study uses one model and one primary generation per eligible task. Because pre-adjudication human labels were not retained, inter-rater agreement cannot be reconstructed and the mechanism analyses remain exploratory. Overall, increasingly successful self-generated tests did not establish semantic correctness, even after excluding unreproduced references.
| Comments: | Accepted at the International Conference on Electrical and Computer Engineering Researches (ICECER 2026) |
| Subjects: | Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.15797 [cs.AI] |
| (or arXiv:2610.15797v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15797 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dishu Yang [view email]
[v1]
Sun, 11 Oct 2026 17:41:15 UTC (76 KB)
来源：arXiv：cs.AI · arxiv.org