arXiv：cs.CL· Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope·· 16 小时前
研究发现基于 LLM 的新颖性评估极不稳定：提示词设计可使准确率波动超 50 个百分点
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
AI 导读
一项受控研究发现，基于 LLM 的新颖性评估极不稳定：仅告知评审认为某一想法新颖、另一想法不新颖，就能让同一想法对的判定超过半数发生翻转，成对准确率波动超 50 个百分点，有时甚至低于随机水平。研究在六个评审模型上测试，发现检索和更大推理预算帮助有限，两个专门构建的新颖性评估器还不如最便宜的提示词基线。该结果对自动化创意系统所报告的新颖性提升提出质疑。
正文
Abstract:AI-based ideation systems are often evaluated on the novelty of their ideas, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that seemingly small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.02022 [cs.CL] |
| (or arXiv:2610.02022v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02022 arXiv-issued DOI via DataCite |
Submission history
From: Noy Sternlicht [view email]
[v1]
Thu, 1 Oct 2026 16:43:18 UTC (1,095 KB)
[v2]
Wed, 14 Oct 2026 17:45:03 UTC (1,034 KB)
来源：arXiv：cs.CL · arxiv.org