arXiv：cs.CL· Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie·· 4 小时前
LintEval：在 Linter 引导的文件重构上评测 LLM
LintEval: Benchmarking LLMs on Linter-Guided File Refactoring
AI 导读
研究者提出多语言基准 LintEval，用于评测 LLM 在 Linter 引导下的文件重构能力，覆盖 C++、Python 和 Java，包含 13,061 个程序与 171,662 条 lint 告警。
正文
Abstract:Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other patterns in source files. However, existing code benchmarks do not explicitly evaluate linter-guided file refactoring, where models must reduce lint alerts in source files with correct initial functionality while satisfying functionality preservation. We introduce LintEval, a multilingual benchmark for linter-guided file refactoring across C++, Python, and Java, comprising 13,061 programs with correct initial functionality and 171,662 lint alerts extracted using language-specific linters. LintEval jointly evaluates functionality preservation, lint alert reduction, new alert avoidance, and edit minimality. Experiments reveal a persistent trade-off: closed-source commercial models reach up to 90.4% Reduction@P but achieve only 67.1%-68.3% Pass@1, while several open-source small models show low or negative Reduction@P. A controlled fine-tuning study further shows that supervision format matters: refactoring-oriented SFT raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%. LintEval enables controlled evaluation of models that improve source files under functionality preservation.
| Comments: | Accepted to Findings of EMNLP 2026. Code available at this https URL |
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.20027 [cs.SE] |
| (or arXiv:2610.20027v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20027 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tongwei Deng [view email]
[v1]
Wed, 14 Oct 2026 11:36:55 UTC (498 KB)
来源：arXiv：cs.CL · arxiv.org