arXiv：cs.CL· Anubhav Jangra, Smaranda Muresan·· 16 小时前
DivAlgo：代码生成模型算法多样性系统评估基准
DivAlgo: A systematic evaluation of algorithmic diversity for code generation models
AI 导读
研究者提出 DivAlgo 基准，用于评估代码生成模型的算法多样性，并引入轻量级相似度指标支持大规模评测。对 24 个现代代码生成模型的实验显示，模型无控制生成的代码多样性仅为人类的约 31%，最佳多样性生成策略也只能达到人类约 70%。模型虽可被引导按人类策略生成解法，却无法复现人类解法的多样性。
正文
Abstract:Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to enable large-scale evaluation. Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing ~31% of human diversity), and even the best diverse code generation strategy could achieve only ~70% of human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models in capturing the algorithmic diversity humans exhibit, and that overexposure to model-generated code can narrow programmers' algorithmic vocabulary.
| Comments: | 24 pages, 18 figures |
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19157 [cs.SE] |
| (or arXiv:2610.19157v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19157 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anubhav Jangra [view email]
[v1]
Tue, 13 Oct 2026 20:24:42 UTC (611 KB)
来源：arXiv：cs.CL · arxiv.org