arXiv：cs.CL· Ruchira Dhar, Qiwei Peng, Anders S{\o}gaard·· 16 小时前
语言模型评测方法论面临哪些挑战：一项跨越四十年的综述
On Challenges in Evaluation Methodology: A Review
AI 导读
一项跨越四十年的范围综述梳理出语言模型评测方法论的 22 类反复出现的关切，并发现部分关切在近期文献中关注度下降，且许多未出现在机构检查清单中。该研究为研究社区重新审视评测方法和开展领域级干预提供了参考。
正文
Abstract:Recent advances in language models have prompted a growing body of work that questions prevailing evaluation practices. However, many such critiques have already been discussed in natural language processing (NLP), a field with a long history of methodological reflection on evaluation. We conduct a scoping review spanning four decades and develop a taxonomy of 22 recurring concerns. For each concern, we examine how recently it was discussed in the reviewed literature and whether it receives explicit attention in institutional checklists, finding that some concerns have become less visible in literature and many lack coverage in these checklists. Our work provides an overview of the major concerns in evaluation methodology that have shaped the field and identifies potential targets for renewed investigation and field-level intervention by the research community.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2604.25923 [cs.CL] |
| (or arXiv:2604.25923v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25923 arXiv-issued DOI via DataCite |
Submission history
From: Ruchira Dhar [view email]
[v1]
Wed, 1 Apr 2026 10:29:43 UTC (911 KB)
[v2]
Wed, 14 Oct 2026 06:43:26 UTC (306 KB)
来源：arXiv：cs.CL · arxiv.org