arXiv：cs.AI· HyunJoon Jung, William Na·· 18 小时前
LLM 智能体评审员的分数与覆盖度解耦：基于人格的评审面板何时可信、需要多少人
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
AI 导读
一项针对 LLM 智能体评审员的研究用 15 项任务、两组模型配对共 960 次会话验证：基于人格的智能体评审在图灵式检验中与人类评分者难以区分。研究还发现分数-覆盖度解耦现象——质量分数随评审面板规模呈对数增长，而独立问题发现数呈次线性幂律增长，分数饱和速度约为发现速度的两倍。消融实验表明，产生这种规模化特性依赖结构化人格条件设置，而非简单提示词。
正文
Abstract:LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.
| Comments: | Accepted to COLM 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2604.00477 [cs.AI] |
| (or arXiv:2604.00477v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.00477 arXiv-issued DOI via DataCite |
Submission history
From: HyunJoon Jung [view email]
[v1]
Wed, 1 Apr 2026 04:44:21 UTC (22,089 KB)
[v2]
Sat, 10 Oct 2026 03:38:36 UTC (3,020 KB)
来源：arXiv：cs.AI · arxiv.org