如何衡量前沿 LLM 在自动化研究中的创造力

This paper has been withdrawn by Mengzhuo Chen

No PDF available, click to view other formats

Abstract:Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. We propose a set of metrics to evaluate creativity along the two dimensions of valueness and novelty. Valueness assesses whether each proposed idea is useful, while novelty is evaluated from three perspectives: whether the same idea has appeared before (Exact-Match P-Novelty), whether the modified variable or variable combination has been explored before (Variable-level P-Novelty), which reflects the breadth of research-space exploration, and whether the proposed idea is explicitly attributed to external knowledge in the model's reasoning (H-Novelty). Our evaluation shows that the models achieve relatively similar Valueness and Exact-Match P-Novelty scores, while differing substantially in Variable-level P-Novelty. H-Novelty is also consistently high among the models for which it can be evaluated. Notably, further correlation and idea-level performance analyses reveal a strong positive correlation between Variable-level P-Novelty and research performance.

Comments:	After discussion with our advisor, we concluded that the manuscript requires substantial further improvement before being made publicly available. As these revisions are expected to take a considerable amount of time, we would like to withdraw the current version for now and resubmit a more complete and robust version once the work has been substantially improved
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:2609.14057 [cs.CL]
	(or arXiv:2609.14057v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.14057 arXiv-issued DOI via DataCite

Submission history

From: Mengzhuo Chen [view email]
[v1] Sat, 12 Sep 2026 17:15:09 UTC (49 KB)
[v2] Tue, 22 Sep 2026 12:44:12 UTC (50 KB)
[v3] Wed, 7 Oct 2026 14:58:16 UTC (1 KB) (withdrawn)